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Introduction 



Schools are not as dangerous and vulnerable as we believe, 
despite indicents like Columbine, which are almost apocalyptic in 
intent and outcome. To dispel myths about an epidemic of school 
violence, without ignoring that it is pervasive, we need to think 
about school violence more evenly. In the past few years, as we have 
learned more about the incidence and causes of school violence, we 
have developed more sophisticated approaches for preventing it and 
ameliorating its effect on its victims and bystanders, and perpetra- 
tors, alike. But not enough is known yet about the value and impact 
of these efforts. 

Schools have come far in finding the right programs to deal 
with a variety of problems in preventing school violence. Daniel 
Flannery, one of the authors of two papers in this collection, nicely 
describes them in another paper: 

Some focus on working with individual children identified 
by teachers or peers as aggressive or at risk for school fail- 
ure. Others combine a focus on individual and family risk 
by integrating school-based programs and work with par- 
ents and families, peers, or community members. Still other 
programs integrate an individual risk focus with attempts 
to change the school environment. Most strive to increase 
student social competence and to reduce aggressive behav- 
ior. (Flannery, 1999, p. 1) 

All of these programs recognize that developing students in 
families, schools, neighborhoods, and in a society that enhances 
opportunity and resilience and reduces risk, is the best prevention 
and antidote to school violence. This is why the best anti-violence 
approach will engage and integrate many home and community life 
influences. 

As educators we feel the imperative to reduce school violence, 
and have the ingenuity to conceive, design, and implement a variety 
of programs, some research-based, some intuitively sensible; how- 
ever, we do not have enough empirical evideiice that these interven- 
tions are successful. There are many reasons why this has happened. 
Most educators are more interested in practice than understanding 
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it, and are not prepared to study it, even if they are interested. But 
equally important, there are scarce resources for evaluations that are 
meaningful to program staff and that can add to our knowledge 
about successful school anti-violence practices. We have not been 
able to train staff either to conduct an internal evaluation, to use an 
outside evaluator, or to create a cost effective and efficient collabo- 
ration between them. In compiling the essays in this collection we 
have tried to show how these problems can be overcome in practice, 
without minimizing or whitewashing the extent or depth of the 
constraints. 

We have arrived at the moment that we can no longer support 
programs, including school violence prevention programs, without 
some reliable outcome data, at least data demonstrating their 
promise. But the burden for evaluation should not be the program 
official’s alone. It has to be shared by funders, who can help local 
programs evaluate their efforts, for their own improvement. These 
evaluations can also help the funders themselves wisely invest in 
future efforts. Beyond knowing whether their money has been well- 
spent, funders need to use the knowledge gained from program 
evaluations to determine whether their grants have brought about 
the social changes that they advocate. 

Funders ask their grantees to use accepted social science meth- 
ods to determine which of their objectives they have achieved (if 
any), why, and how. But although evaluation is a condition of many 
grant awards, it has been loosely monitored and only intermittently 
honored. Most funders have not been able to structure a grant 
award to increase the likelihood that the program will be evaluated, 
just as most grantees do not work in a culture that supports pro- 
gram evaluation; If funders seriously want grantees to assess how 
their performance has met their objectives or plans, they must 
actively help grantees to create this culture. Funders can require that 
grantees demonstrate greater organizational capacity, particularly 
the leadership and staff skills necessary for logical planning, moni- 
toring, and evaluation. Most small-scale local programs, however, 
will not have this capacity, and will not be able to create it, without 
the help of the funder, especially community-based organizations. 
This means that an award should provide additional funds, to be set 
aside for the leadership development and staff training that will lead 
2 to better program evaluation. 
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But the incentive of financial support is not enough. Funders 
can make available information about the growing number of non- 
technical guides to the basic principles of an internal program eval- 
uation, which can help staff in program definition, planning, imple- 
mentation, continuous improvement, as well as in measuring pro- 
gram outcomes. The essays in this collection review many of these 
resources, and themselves provide significant guidance for evaluat- 
ing school violence programs. 

Under ideal circumstances, funders could also help their 
grantees to get outside technical assistance in carrying out a pro- 
gram evaluation, although project staff will stiU need to recognize its 
value. Unfortunately, the cost of the necessary level of technical 
assistance would be prohibitive and drain limited local operating 
funds, if used. There are, however, a number of regional technical 
assistance networks which have devised strategies for providing 
continuing but broad-based assistance. Funders fully committed to 
program improvement and better evaluation should consider sup- 
porting organizations which identify and broker technical assistance 
services expressly tailored to the evaluation needs of local projects. 

Clearly, we can find many efforts to prevent school violence, 
and they are enthusiastically supported locally and nationally. Now 
we have to find out whether they are successful.. 

— Erwin Flaxman 

Director, Institute for Urban and Minority Education and 
the ERIC Clearinghouse on Urban Education 
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Challenges to Evaluating School 
AND Community-Based Violence 
Prevention Programs: 

A Framework For Action 

Daniel J. Flannery and Megan Seaman 



Introduction 

Schools and communities around the country are implement- 
ing violence and substance abuse prevention programs with 
increasing frequency. This is partly due to demands from policy 
makers for comprehensive school safety plans. It is also partly due to 
the perception that violence at school is on the rise, and to the recog- 
nition that schools provide a logical context for implementing com- 
prehensive violence prevention programs. Coupled with the 
increasing frequency of program implementation, funders and pol- 
icymakers have called for more rigorous evaluations of effectiveness 
and outcomes. There is more demand for evidence-based practice, 
and for outcomes which show positive behavior change. While the 
fields of violence prevention and program evaluation have grown 
significantly in the past few years, we are stiU at the beginning stages 
of acquiring rigorous, long-term outcome data on best practice 
strategies for violence prevention. 

Further, there exist few resources for the non-research oriented 
professional seeking to learn about and conduct basic evaluation tasks 
related to program implementation. Not all community-based 
providers or school staffs have access to a university-affiliated 
researcher with expertise in evaluating violence prevention programs 
in apphed settings. Not all researchers have experience in conducting 
applied research in schools and communities. And few policy makers 
understand the complexities of doing research in the arena of violence 
prevention in a way that allows research and data to be applied to 
everyday practice and policy. 

The purpose of this essay is twofold: (1) to provide guidance on 
developing an infrastructure for the evaluation of violence prevention 
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programs, and (2) to discuss some of the challenges and barriers to 
evaluating school and community-based violence prevention pro- 
grams. We begin with a brief overview of the different types of evalu- 
ations that can be conducted and how each might inform program- 
matic issues, intervention, or outcomes. We then review the benefits 
of conducting a program evaluation and the basic steps in doing so. 
The next section presents some challenges to evaluating school and 
community-based violence prevention initiatives and, where appro- 
priate, some guidance on how to manage these challenges. The infor- 
mation presented here builds upon earlier work in this area (Flannery, 
1997; 1998). This overview is not meant to be comprehensive or 
exhaustive; rather, this is a summary of the elements of an effective 
infrastructure for evaluation and of the practical challenges of pro- 
gram implementation and evaluation likely to be faced. 

Types of EvAmmoN 

In any intervention program, four basic questions are asked: 

( 1 ) What kind of intervention is needed cind who should be targeted? 

(2) What are the program’s desired results and what will be changed? 

(3) What components of the program make it successful? and 

(4) Is the program cost effective? 

Recently, funders at the local and Federal levels also have been 
asking another question: What is the evidence that the intervention 
can be sustained locally over the long term? 

Answering these questions requires paying attention to the 
development of the program’s infrastructure for evaluation, pro- 
gram planning, and long-term implementation. 

Four basic types of evaluation can be integrated into the exist- 
ing structure of most schools or community-based programs to 
address these questions. Programs may not have the need to con- 
duct each type of evaluation, but they should be aware of their 
options and the role that each type can play to help them, answer the 
most basic questions about their intervention. The four types of eval- 
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uation strategies most commonly utilized are needs assessment, 
outcome evaluation, process or monitoring evaluation, and cost- 
benefit analysis. 

Needs Assessment 

A needs assessment (or formative evaluation) helps an organi- 
zation determine what it needs to do regarding violence reduction 
and prevention. Some organizations may skip a needs assessment, 
either due to a perceived lack of resources or because they believe 
that solely a commitment to do something to reduce violence is suf- 
ficient to move forward. This could be a costly mistake in the long 
run. A needs assessment can provide an accurate description of the 
problem(s) that the community faces. Information needed to 
describe the problertl can come from many sources, including avail- 
able archival information, from local or national surveys of stu- 
dents, school staff, community providers, or parents; or from focus 
groups or interviews with community members. Examples of com- 
munity-based archival data that may be helpful in a needs assess- 
ment include: police records; hospital records on emergency room 
referrals or injury rates; school nurse records on fighting and relat- 
ed injuries; school records on attendance, discipline infractions, or 
weapons violations; outpatient mental health records on incidence 
of mental health problems; and Federal or foundation-based sur- 
veys that include local data. A needs assessment can also do the fol- 
lowing (Thornton, Craft, Dahlberg, Lynch, & Baer, 2000); 

(1) Identify the intended participants for an intervention. Is the tar- 
get population youth, families, or both? Is there a need for universal 
preventive intervention where all youth in a school or community 
are exposed to a program, or for more targeted services to at-risk 
youth? Win the program focus on young people in elementary 
school or on all youth through high school? 

(2) Identify possible settings in which to reach intended partici- 
pants. Many different settings exist for interventions to prevent vio- 
lence. Where the intervention occurs will impact evaluation strate- 
gies, the potential availability of information on program compo- 
nents, and the practicality of gathering information from program 
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participants. A school-based intervention usually has several infor- 
mation streams built into the school system, and is itself a setting 
where information can be gathered from many children and staff in 
a relatively short period of time. If an intervention includes response 
to a crisis in a family’s home, there may be greater restrictions on the 
information-gathering process. 

i'5) Analyze the role of violence in the community. Development of a 
customized, and therefore more effective, long-term violence pre- 
vention strategy requires consideration of the following: (a) the 
nature and prevalence of violence perpetration and victimization in 
the school or community; (b) the precursors to youth violence which 
affect the children in the school or community; and (c) the impact of 
violence on child adjustment, mental health, and learning.. 

(4) Set goals and objectives. The first question an evaluator should 
ask of a program administrator is “What is the goal of the pro- 
gram?” or “What do you want to see changed as a result of doing this 
intervention?” If the goals of an intervention are not clear, then the 
evaluation plan will not be as clear or effective. At the program level, 
goals should be consistent with the behaviors and settings targeted 
for intervention. Goals should be broad statements about what the 
program seeks to achieve, accompanied by specific objectives that 
reflect how the program is going to reach the goals. A common pit- 
fall occurs when an agency sets extremely lofty goals with unobtain- 
able or unrealistic objectives. Aspiring to service 90 percent of an 
identified high-risk target population may be admirable but not 
achievable. Similarly, reducing violence rates by 50 percent in a 
school may be overly optimistic, at least in the short term. 
Ambitious goals may help a program receive initial funding, but the 
program may be setting itself up for failure simply because its goals 
and objectives are not realistic to the task at hand. Unreachable goals 
also set up the evaluation for failure as the conclusion may well be 
that initial goals were not achieved. Such an evaluation would be 
inaccurate if the setting and circumstances of the intervention or 
population made those goals unattainable from the beginning 

(5) Select an appropriate program or intervention strategy (or mul- 
tiple programs) to address desired goals and objectives and meet the 7 
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needs of participants. This may be one of the most daunting tasks 
that an organization newly embarking on a violence prevention ini- 
tiative may face. There are now a multitude of guides on school and 
community-based violence prevention. Most guides provide 
descriptive information about the types of programs available, but 
few provide a framework for how to evaluate the program effective- 
ly or how to evaluate one program relative to another when deter- 
mining which would best fit particular local needs. Choice of inter- 
vention strategy (e.g., school or community-based) will have major 
implications for how that program can or cannot be evaluated. 

Addressing each of these needs assessment tasks efficiently and 
comprehensively will contribute to the utility and efficiency of the 
evaluation strategy, and will increase the potential for meaningful 
results. It is important to keep in mind that an intervention plan that 
win work in a larger community or in an urban setting may not be as 
effective or necessary in a smaller or rural community. Interventions 
that are imposed on a community by an outside entity are less likely 
to succeed than locally-generated or -supported initiatives. 

Outcome Evaluation 

The second type of evaluation, an outcome evaluation, answers 
the question “what changed because of the intervention?” Did the 
program reduce child aggressive behavior or violence? Did the pro- 
gram result in improved social skills or an increase in school atten- 
dance? The outcome questions addressed in the evaluation should 
flow directly from the program goals and objectives. The types of 
outcomes sought for an intervention will also impact the types of 
evaluation methods used to assess them. Some outcomes can be 
measured using available archival data, others by surveys already 
administered to participants in community-based agencies or to 
students at school. Still other outcomes may be more readily 
assessed by direct observations of behavior, or gleaned from focus 
group discussions of a participant’s experience in a program. 

When selecting behavioral outcomes, it is important to remember 
that common factors underlie a constellation of problem behaviors, 
including violence and substance use. For example, the National 
Longitudinal Study of Adolescent Health has consistently demonstrat- 
8 ed the role of attachment to school and school achievement as protec- 
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tive factors which reduce the likelihood that an adolescent will engage 
in delinquent or violent behavior and substance use (Resnick et al., 
1997). If these outcomes are identified outcomes of a violence preven- 
tion initiative, they must be clearly explicated, and the rationale for 
how they will be realized based on exposure to the intervention, clear- 
ly developed. An outcome may be a desirable behavior, but if it is not 
clearly linked to the intervention, and is not defined in a way that it is 
measurable, then program goals will not be met and the evaluation of 
outcomes will be unrelated to program content. Clearly defining pro- 
gram goals, desired outcomes, and their connection will go a long way 
toward establishing relevant and effective outcome assessments of the 
program’s success as well as helping to identify possible limitations. 



Process Evaluation 

Process evaluation (also referred to as program monitoring) 
addresses the question “what makes the program work and why?” One 
of the most difficult aspects of an evaluation is determination of which 
specific components of an intervention are related to which specific 
outcomes. And this becomes increasingly difficult as interventions 
become more comprehensive, multi-level, and complex. Children are 
exposed to a myriad of influences in their schools and communities 
that may be related to behavior outcomes, and isolating the effects of a 
young person’s exposure to a specific intervention may be next to 
impossible. This is where a more qualitative approach to evaluating 
program effects may be helpful. Sometimes program participants can 
say which of their experiences were impactful. These impressions are 
sometimes more difficult to glean from a quantitative survey. 

Another often overlooked component to process evaluation is 
an assessment of the elements of a program or intervention. Is the 
program being implemented according to the original plan? Does 
the intensity of the program change over time? Does staff imple- 
ment the program differently over time? All of these factors related 
to the process of program implementation have major implications 
for program effectiveness and outcomes. Fidelity of program imple- 
mentation is critical to intervention success. Even the best, most 
well-intended program will not have its intended effect if it is not 
implemented with fidelity and by well-trained and supported staff. 9 
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A cost-benefit evaluation answers the question “is the program 
cost effective?” It might include an assessment of how much the 
program costs to implement (per individual or per school), or how 
much the program saves in other related costs (e.g., diversion to 
incarceration or keeping a child in a regular vs. an alternative school 
setting). Costs related to implementing a program can include direct 
program outlays for materials and training. They may also include 
the costs of staff time for participating in training, implementing 
the intervention (this can be problematic if it takes away from aca- 
demic instruction time, given the emphasis in some districts on 
achievement test scores), and collecting evaluative or program data 
from participants or records; and of ancillary services included in 
the program initiative (e.g., outpatient therapy for families identi- 
fied through a school-based initiative). 

The Evaluation Process 
Benefits of an Evaluation 

Why should a school or community-based program conduct 
any kind of evaluation, especially if it is implementing one of the 
identified “best practices” previously evaluated by some other group? 
One must keep in mind that there are lots of types of evaluations that 
can be conducted, at lots of different levels. Evaluation of a previous- 
ly tested best practice program may be more limited in scope (and 
thus in cost) than evaluation of a newly created or more complex 
intervention (e.g., a combination of many programs). Some of the 
benefits of conducting evaluation include the following: 

Understanding of the population served. Every school and com- 
munity is different. The participants in any program wUl vary by 
ethnicity, age, family structure, neighborhood, socioeconomic status 
(SES), and their set of previous experiences. The impact of the inter- 
vention on the target population will be different from the effects 
found in other schools or communities. 

Understanding of the services provided. What is developed on 
paper is seldom a complete reflection of what happens in reality. Even 




the best designed program will work a little differently in one com- 
munity or school than it did in another location. Understanding how 
a program works for the target population in the setting identified for 
intervention, and assessing its impact on policy in the local commu- 
nity, are important pieces of information that should continue to 
guide decisions about implementation, evaluation, and funding. 

Improvement of the services. Ideally, evaluative data should be 
provided back to the program on a regular basis so that information 
gleaned from the research can be used to modify or improve pro- 
gram services. If an assessment of program participants shows that 
they are dissatisfied with a component of the service, then modifi- 
cations to that corhponent should be considered. At a minimum, the 
reasons for the dissatisfaction should be explored. Changes to the 
. intervention, based on data, should be considered before the end of 
the intervention, especially if the program is set up to last several 
years. No one can afford to wait several years for data to show 
whether a program is effective. 

Data to report to collaborators. Violence prevention initiatives 
are increasingly collaborative, multi-systemic, and complex. For 
example, the Safe Schools/Healthy Students initiative funded by the 
U.S. Departments of Education, Justice, and Health and Human 
Services requires collaboration among schools, mental health care 
providers, law enforcement, community organizations, and evalua- 
tion researchers. Effectiveness of an intervention requires active par- 
ticipation by all collaborators because one partner’s efforts may 
directly impact on outcomes achievable by another partner. 

For example, if an intervention to reduce childhood trauma 
symptoms experienced because of exposure to violence depends on 
police response and referral to the program, it would be important 
to be able to provide evidence of the effectiveness of the interven- 
tion to police officers who initially respond to violent incidents. 
Officers are much more likely to continue to make referrals to the 
program if they know that it benefits the children and families, or if 
referred families make fewer calls to the police for assistance after 
participating in the intervention. Having police agree about the ben- 
efits of a mental health intervention would also help the program 
obtain funding for services. 
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Data to support funding of the program. In an era of increasing 
competition for more limited resources, programs have the burden 
of providing objective data demonstrating effectiveness. Funders are 
inundated with requests for support. Programs that have quality 
outcome data are in a better position to obtain those limited 
resources than programs without any evidence of effectiveness. 

Enhanced scientific knowledge. Despite the growth in the num- 
ber of school and community-based violence prevention programs, 
the field still lacks the scientific knowledge base that accrues with 
well-designed evaluation studies of program effectiveness. 
Longitudinal studies are particularly sparse (Drug Strategies, 1998). 

Requirements for an Evaluation 

One of the most critical elements of doing effective evaluation 
is the estabUshment of an adequate system for information gather- 
ing. This includes developing procedures and mechanisms for gath- 
ering data, having adequate access to information, and managing 
that information so that it is usable for the program evaluation. 
Information gleaned should be available for sharing across systems 
(e.g., schools with juvenile courts and with children and family ser- 
vices) to enhance the efficiency of delivery of services and to enable 
outcomes to be tracked across settings. Many children, especially 
targeted, at-risk youth, are involved in multiple systems. Their expo- 
sure to a school or community-based intervention may affect out- 
comes in any one of those other systems. 

It is important not to underestimate the need for adequate 
computing power (and expertise). Some communities have well- 
established infi'astructures for information gathering and manage- 
ment in their schools, health departments, police departments, and 
community organizations. Other communities still collect much 
information “by hand” via paper and pencil. If vital records are only 
available through a manual search of individual files, the cost and 
complexity of using that data in an evaluation could be significant- 
ly increased. Some communities have records stored in computer 
systems, but the access to those systems is cumbersome and time 
consuming. For example, access to some juvenile court records 
12 requires a person to uje a dedicated computer terrhinal or data line 
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and to type in individual names to determine adjudication status. 
Thus, if the effectiveness of a diversion program that serves several 
thousand youth annually is being evaluated, gathering the informa- 
tion (and reentering it into the program’s own data base system for 
analysis) can be time consuming and costly. 

Decisions about intervention strategies and effects should be 
based on local information and practice. It is always useful to under- 
stand the information already in hand before proceeding with new 
data gathering or intervention planning. Existing information from 
the targeted population, school, or community can serve as baseline 
data. Baseline data provide information from which any measurable 
change that results from an intervention can be assessed. 

One of the most challenging tasks in implementing and evalu- 
ating a violence prevention program is the development of the 
infrastructure for sustainability of program implementation. Few 
funding streams are available that will provide resources for more 
than a few years, even for the most promising, well-evaluated pro- 
grams. The reality, however, is that few large-scale interventions are 
maintained in any community without significant external support 
and resources. The challenge is to work from the start to garner 
political and community support for the sustainability of the pro- 
gram’s operation. The more comprehensive the effort, with multiple 
providers and multiple systems benefiting from the service, the 
more likely the program will be institutionalized and supported 
over the long term. Good evaluation data on implementation, sus- 
tainability (process) and outcomes can contribute significantly to 
this effort. 

Steps in Conpucting an Evaluation 

There are several basic steps or components to conducting any 
evaluation. Within the context of agreeing that evaluative efforts can 
be more or less comprehensive (often depending on need or 
resources), the following highlight the basic steps: 

(1) Be clear and specific about the program goals and objectives. 
This first step will drive the development of the rest of the evalua- 
tion design and implementation. One potentially useful evaluative 
tool at this stage is a logic model of the program components. 
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Short-term, intermediate, and long-term goals can be identified. A 
logic model also helps link program components to specific out- 
comes, part of the process evaluation. The Kellogg Foundation 
Evaluation Handbook (1998) has a helpful section on the develop- 
ment of logic models that is easily understood by the non- 
researcher. Logic models also help to convey program elements and 
expectations in a simple but comprehensive manner to potential 
funders and policy makers. It is important to keep in mind the 
answer to the question "What do you want to find out or show when 
you’re done?” 

It may also be helpful to distinguish, from the beginning, what 
the program needs to change versus what it wants to change. Need 
may be determined by the initial needs assessment or requirements 
of the funder or political wiU. Want may be more idealistic or long 
term, but it is important to consider these questions up front, as 
need often translates later into want, especially when the evaluator 
is explaining to the funder or politician the reality of what was 
accomplished versus what someone wants to see accomplished. 

(2) Decide how to determine whether the intervention works. When 
will it be apparent that the program has demonstrated something 
positive or worthwhile? There are statistical measures of effects. For 
example, comparing a group that received program services to a 
group that did not, an evaluator could compare rates of problem 
behaviors before and after program implementation. If the rate of 
the group that received the intervention is lower than that of the 
comparison group by a statistically significant margin, it might be 
concluded that the treatment had the desired effect. 

Is there any other evidence for effectiveness? The answer partly 
depends on the original goals and how outcomes were defined. Is 
the focus on behavior change, cost effectiveness, the program’s gen- 
eralizability to other groups or communities, or all of the above? 
The answer also depends on who is asking. A legislator may want 
evidence of overall declines in crime rates for the community, while 
a school administrator may want specific data on reductions in indi- 
vidual student discipline incidents, while a foundation officer may 
want to see evidence of effective collaboration, with efforts by one 
partner impacting the need for resources by another partner to pro- 
vide services. 




(3) Determine the scope of the evaluation. The scope of the evalua- 
tion will depend on several factors. First, the nature and extent of 
the intervention will determine whether the evaluation is compre- 
hensive or more limited. Second, resources will matter. It is typical 
for 10 to 15 percent of a projects total budget to be allocated for 
evaluation. The amount will be closer to 10 percent if the interven- 
tion has been previously evaluated in other settings, if evaluation 
tools have already been developed or identified firom existing mea- 
sures, and if procedures for information gathering are in place. 
Assuming these conditions, and that information is readily accessi- 
ble from several sources and systems (e.g., archival school data are 
computerized and available), then costs related to evaluation can be 
minimized. If, however, none of these conditions exist, and if the 
intervention is multi-level and complex, spanning multiple systems, 
then evaluation costs will be higher, perhaps even exceeding 15 per- 
cent of the total project costs. If evaluation team members are 
expected to gather information directly via survey or observation^ or 
if additional staff is required to conduct archival record review, these 
labor intensive activities will require more resources to complete. 

Within the scope of a large intervention project, it is possible to 
prioritize evaluation tasks and to limit the focus to a few major eval- 
uative questions. The project may decide that the outcome ques- 
tions are a priority, and that cost-benefit analysis may be undertak- 
en at another time. If a “best practice” intervention is being imple- 
mented, whose components and effective mechanisms are clearly 
identifiable, then there may be less emphasis on conducting a thor- 
ough process evaluation. In the end, it is best to conduct a few eval- 
uative tasks well than to try to do a comprehensive evaluation of 
every type for every component of the program, and do them poor- 
ly. When the project is completed, administrators want to have 
something reliable and valid to say, even if they cannot answer every 
question asked at the beginning of the program. 

Whatever the size of the evaluative effort, it is important to 
think along the lines of developing a multi-trait, multi-method 
matrix. This means that from an evaluation perspective, the pro- 
gram will benefit most from a focus on gathering different types of 
information from as many different types pf sources as possible. For 
example, while the intervention may be focused on violence pre- 
vention, it may be very helpful to measure aggressive and delinquent 
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behavior (multiple traits) in addition to violent behavior, because 
actual violence may be perpetrated by a very small portion of the 
target sample. Similarly, it may be worthwhile to consider measur- 
ing social skills or social competence, and more positive behaviors 
that rnay increase due to the program, rather than just focusing on 
reductions in negative behavior as indicators of program success. 
Regarding multiple methods, where possible an evaluation design 
should include gathering information via a variety of available tech- 
niques, including opinion or experience surveys, collection of 
archival data or review of records, interviews, behavior observa- 
tions, and focus groups. The method utilized will depend on prag- 
matic issues of time, cost, staffing requirements to gather the infor- 
mation, and general feasibility. It may not be feasible to conduct 
focus groups or extensive interviews of participants, but it may be 
feasible to survey a sample of program participants before and after 
their participation in the intervention. 

Information should also be gathered from a variety of sources, . 
whenever possible: from the target population (e.'g., children), those 
who provide the intervention (e.g., teachers), those who may bene- 
fit from or participate indirectly in a program (e.g., principals, 
police officers), and others associated with the intervention or pop- 
ulation (e.g., parents, community mental health care providers). 

When conducting any type of program evaluation, it is impor- 
tant to appropriately define or operationalize the constructs or behav- 
iors targeted for change. There should be strong consideration of this 
process, especially if information is going to be gathered at more than 
one point in time. Once collection of information begins (i.e., at base- 
line), the research design and the applicability and utility of the find- 
ings will be compromised if the assessment measures change. 

The simplest route to take is to use instruments that have been 
previously used and validated on a similar group of program par- 
ticipants. Developmen tally appropriate questions are particularly 
important to pay attention to. Questions about violence designed 
for high school students will probably not be appropriate for ele- 
mentary school children. Questions should also reliably assess the 
targeted behaviors. If, for example, the program focus is on aggres- 
sive behaviors, surveys or instruments that are valid and reliable 
measures of aggression should be selected rather than something 
else. A related question to consider is whether multiple informants 
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would be rating the targeted behavior consistently over time and in 
the same way for different participants. There are many validated 
and reliable surveys and scales already in existence. Some may have 
been used previously to evaluate the program. Some may be 
reviewed in other publications (e.g., Thornton et al., 2000). The 
consistent mantra is that an evaluation should start with resources 
in hand, then consult existing projects and literature before devel- 
opment of new assessment tools is begun. Modifying an existing 
survey (for content, length, developmental appropriateness) is 
almost certainly going to be easier and more cost effective than 
developing a new survey. 

(4) Decide on the research design. What should be the strategy for 
collecting information on program effectiveness? There are three 
basic components to any evaluation effort that will make the results 
more readily interpretable and valid. The first is collection of out- 
come data before the intervention is implemented. This provides 
baseline information from which change can be assessed. The second 
is assessment, whenever possible, of a comparison group of individ- 
uals not exposed to the intervention. A comparison group should be 
as similar as possible to the individuals who participate in the pro- 
gram. Similarities in terms of the factors that can affect behavior 
change and outcome are particularly desirable to achieve. For exam- 
ple, if the population targeted for the intervention is families in the 
neighborhood that live below the poverty level, then the comparison 
group would ideally be comprised of other similar families who may 
be eligible for the program but for some reason (which is docu- 
mented) do not participate. If the comparison group is similar on 
key factors (neighborhood, SES, child gender, family size, parent edu- 
cation level, ethnicity), then differences between the groups seen after 
the intervention are less likely to be attributable to those factors and 
more likely to be due to participation in the program. 

The third basic component, and the hallmark of any rigorous 
evaluation study, is the random assignment of individuals to the 
intervention program group or a nonintervention comparison or 
control group. This standard is the most difficult to achieve, both 
practically and ethically, and is rarely achievable in schools and. 
communities. Let’s return to the earlier example of a community- 
based intervention where police officers refer children and families 
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exposed to violence to a crisis intervention team for assessment and 
further services. How practical or ethical would it be for a police offi- 
cer or crisis worker to show up at a home immediately after a trau- 
matic incident has occurred and offer; “We know that you may want 
these services and we believe that you may benefit from them, but 
would you be willing to be randomly chosen either to get services or 
be in a no-service comparison group?” This would be neither feasi- 
ble nor realistic. How many high-risk, difficult to engage families 
would agree to those conditions? How many officers or crisis work- 
ers would agree to propose those conditions to families in those cir- 
cumstances? Not many. It is always important to balance the research 
design and desire for rigorous evaluation with the pragmatics of ser- 
vice provision and ethical, fair treatment of program participants. 

There are some programs that, on the surface, may not appear 
to lend themselves to a randomized design or to the identification of 
a control or comparison group. The family-based intervention 
Families and Schools Together provides services to about 15 families 
at a time (per 8 -week cycle) at each site, 'typically a school 
(McDonald & Frey, 1999). Sometimes, more than 15 families 
express an interest in participating. Depending on how families are 
identified, and the number of 8-week cycles that the program can 
afford to implement, it may be possible to assign families originally 
interested in the program randomly to an immediate intervention 
or a future program cycle. This design permits some comparison of 
intervention vs. nonintervention control groups, but also eventual- 
ly provides the service to all eligible families that want them. The 
necessary baseline or pretreatment data could be collected from all 
families before they are randomly assigned to a program cycle, 
assuring that there is comparable information from all eligible fam- 
ilies. If the families all come from the same school and neighbor- 
hood, then they will be similar on these important factors, and dif- 
ferences observed would more likely be attributable to participation 
in the program. 

(5) Implement the evaluation. Remember the point made earlier: 
in implementing an evaluation, it is better to do a few things well 
than to do many things poorly. When the evaluation is completed, 
it is crucial to have something to say with confidence, based on reli- 
able and valid information gathered from multiple sources using 




multiple methods. The first task before implementation is to revisit 
the original research questions to make sure that all of the work 
done in preparation for the evaluation has not strayed from assess- 
ment of the original program goals and objectives. If the methods 
and instruments to specific questions or program components can- 
not be connected, it should be determined whether essential infor- 
mation was collected. This “extraneous” information should not be 
completely discounted, however. Many times we have conducted a 
program evaluation and found that information gathered because 
we thought it might be interesting or helpful (not necessarily explic- 
itly tied to our original research questions) turned out to be essen- 
tial in interpreting our main findings or to lead to additional impor- 
tant questions about the program or participants that we never 
anticipated before we started. So, it is useful to be creative, flexible, 
and as comprehensive as possible without overwhelming the pro- 
gram staff, the evaluation staff, or the resources available for the 
overall project. 

Once everyone is comfortable with the design of the evaluation 
and with the instruments and methods for gathering information, 
the next step is to begin collecting the data. The first category of 
information, to be collected before services are provided, is baseline 
data on program participants (and individuals in the comparison 
group). Optimally, the evaluator has been “at the table” from the 
beginning so that questions about the evaluation design, instru- 
mentation, and method of information gathering can be answered 
by program staff before anything unexpected occurs. For example, 
an evaluation design may call for gathering archival data, with the 
assumption that a community-based agency will have basic demo- 
graphic information available on participants in cornputer files. The 
evaluator may assume these data are accessible and in a form 
amenable to a merge with other survey data collected, and can be 
analyzed using a standard statistical analysis program. Of course, 
this is often not the case, and the evaluator may find that while the 
data are in a computer file, the information desired may be difficult 
to obtain, protected by agency confidentiality rules, or be housed in 
a data management system that is not easy to access for detailed 
analysis. The sooner these challenges are identified, the more likely 
a reasonable solution (involving resources, staffing, or evaluation 
priority) can be found. 




Sometimes the evaluation is occurring while the program con- 
tinues to be developed and implemented. This tends to be truer for 
larger, multi-component community-based projects. It is becoming 
rarer for intervention programs and, therefore, for evaluations to be 
focused on one program that operates independently of other pro- 
grams or influences. These types of projects (e.g.. Safe 
Schools/Healthy Student sites) often combine programs across mul- 
tiple systems, each with its own policies, requirements, data man- 
agement challenges, and program procedures. Trying to merge pro- 
gram elements, information gathering protocols, and procedures 
can be a daunting task. We find that we often propose gathering 
information that disparate systems all deem important and valu- 
able, but collect differently. Thus, there exists a need to change the 
way individual systems gather or manage that information so that it 
is usable by both the systems and evaluation teams. This sometimes 
requires new information gathering protocols, new consent proce- 
dures for program participants, the development of new forms or 
software, and development of new mechanisms that allow informa- 
tion access and a data format that is usable to individuals outside of 
the original agency. 

(6) Analyze the data. The first task at this stage is to describe what 
was done, and for whom, from a program perspective. This might 
mean documenting the number, characteristics, and type of services 
provided to each participant in the intervention and comparison 
groups. The reality is that not all participants will remain in the 
intervention group, nor will all participants receive aU available ser- 
vices. There maybe important differences between participants who 
received more intervention and those who received less, or between 
participants who agreed to receive services and did so and those 
who also agreed but never took part in more than introductory or 
intake sessions. These intervention group participants may be the 
program dropouts or non -completers. What participant or project 
characteristics distinguished between the dropouts and completers? 
What descriptive information might identify potential barriers to 
program participation? The bottom line is that if it is not known 
who participated in the program, and who did not, then some core 
outcome, process, and cost-benefit evaluation questions cannot be 
answered. 
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The second major component of the data analysis phase is con- 
sideration of the audience for the data analysis. Are the data being 
reported back to project staff? Then the analysis of information may 
focus on program components, who is receiving services, how many 
have completed the program. Or, the focus may be on the com- 
pleteness of the information gathered, or on the utility and ease of 
use of the forms developed for data gathering or consent proce- 
dures. Just placing a question on a form does not guarantee that 
information will be gathered consistently or accurately from referral 
sources, project staff, or program participants. This “process” or 
monitoring issue will directly impact on the amount and accuracy 
of information available to address questions of outcome. 
Consistency, accuracy, and completeness of information being gath- 
ered should be examined early on in a project or during a pilot 
phase so adjustments in protocol, procedure, forms, or processing of 
information can me made. The longer a program waits to do this, 
the more problems will arise later when it is discovered that the 
information anticipated for examination of project goals and objec- 
tives has not been collected. 

If the audience for the evaluation is the program funders, then 
the evaluator may end up analyzing data at a level where the infor- 
mation is more interesting, practicable, and useful for them. How 
many people received services? What is the evidence that the service 
was beneficial, both to individuals and to the community? Positive 
outcomes were realized compared to what or to whom? Funders 
may be interested in the percentage of change over the course of the 
intervention. They may want to know if a change observed is signif- 
icant, but they may be less interested in the statistical procedures 
used than in the way that information is presented. Funders may 
also want to hear about cost issues, as well as long-term sustainabil- 
ity, although we have found it helpful to place our findings in the 
broader context of scientific findings of other local and national ini- 
tiatives. Local funders and policy makers like to see how the project 
they support compares to others around the country. Policy makers 
like to see that they are making a difference, but also how local poli- 
cies reflect national trends. 

The third task for data analysis is review of the program’s orig- 
inal goals, outcome questions, and program monitoring issues. Are 
the data examined in a way that allows all or most of the original 
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questions to be answered? Sometimes there will be data to answer 
part of a question but not aU of it. That is all right. Sometimes data 
analysis leads to unanticipated findings or to questions. We find this 
“data mining” in the analysis phase to be extremely critical and help- 
ful to the overall evaluative process. 

Finally, it is important to be patient. The more data that are 
gathered, and the more complex the intervention, the more involved 
the data analysis phase will be. If information is gathered from mul- 
tiple sources more than once, then longitudinal questions will come 
to bear at some point. These would include questions like “Did pro- 
gram participants change over time?” and “Did degree or amount of 
exposure to the intervention make a difference in outcomes?” The 
latter question may be extremely important to address in the 
absence of a nonintervention control or comparison group. 

(7) Disseminate the findings. Again, remember the audience for the 
evaluation. Schools and community-based organizations are not sci- 
entific journals. Lots of tables with lots of numbers are usually not as 
useful as graphs and figures. Pictures can speak a thousand words. 

Second, briefer is usually better. Few individuals have the time, 
inclination, or need to read long, detailed, scientifically complex 
reports. Longer reports should be generated by the evaluator as 
comprehensive, historical records of the project, data, analysis, and 
findings. However, the original dissemination of project findings 
should probably be presented in executive summary format, with 
the main outcomes summarized in narrative form (with supporting 
graphs or figures) rather than in a long, complex document. The 
fuller report can contain descriptions of the instruments, their reli- 
ability and validity, and the specifics of the data analytic methods 
and procedures. These do not need to be reviewed in the suinmary 
of project goals and findings to members of the community. 

Third, disseminating findings sooner is better than later. The 
evaluator should have the staffing resources to spend adequate time 
cleaning, merging, analyzing, and writing up the data gathered. This 
“end point” is just as important as the need to be clear at the begin- 
ning about project goals and objectives. Why spend several years 
conducting a program and collecting data, but not commit the 
resources for understanding what was found? Further, the data will 
largely drive decisions about continuation of the program. Programs 




are not in a position to wait months or years for evidence of effec- 
tiveness, especially if funding depends on demonstrating promising 
or positive results. At a minimum, it will take an evaluator several 
months from the project end date to analyze data from program par- 
ticipants. This best case scenario also depends on the evaluator doing 
data cleaning and merging on a continuous basis throughout the 
project. Data analysis and preparation of a final report will also 
require the commitment of several research staff, with more staff 
time necessary for bigger, more complex evaluation programs. 

Program Evaluation in School and Community-Based 
Settings and Impucahons for Poucy 

AVAHABIUrY OF RESOURCES 

An evaluation can only be as involved and comprehensive as the 
resources available to support its implementation. This does not 
merely mean the money allocated for evaluation activities, but also 
staff made available to support the evaluation. Usually evaluation is 
one of the first components to be reduced or cut when program 
funding is reduced, but it is the first place that funders and policy 
makers look when considering further support for the program. 
Experienced evaluators can assist a program with utilizing as much 
data as possible to maximize the resources available for data analy- 
sis and dissemination. Some data can be gathered by program staff 
rather than independently by evaluation team members. Some 
methods are more costly than others. Conducting observations is 
usually more costly than analyzing archival data. 

Agency resources and capacity to implement a program and eval- 
uation plan also come into play. They can include administrative sup- 
port, community awareness and support, attention to staff training, 
commitment to the fidelity of program implementation, and the long- 
term sustainability of the program. Few interventions show effects 
after one year of implementation. Recent research also suggests, for 
example, that schools without adequate support and resources to ini- 
tiate a program, including well trained staff, may do more harm than 
good. Doing something is not always better than doing, nothing. 




Concerns about the Evaluator 

What if a university-based research partner cannot be secured? 
While it is better to be physically present at a meeting, and to have a 
firsthand understanding of local concerns, the way organizations 
are structured, and the politics of working together, technology has 
significantly enhanced the ability for researchers to act as consul- 
tants from afar. There are three possible evaluation models for a 
program: (1) use an external, independent evaluator; (2) hire an 
internal evaluator, or use existing program staff to conduct the eval- 
uation; or (3) combine the use of an internal staff person conduct- 
ing the evaluation Avith an external consultant who can provide 
technical expertise on aspects like design, assessment instruments, 
and data analysis. Even if a program is responsible for conducting its 
own program evaluation, there now exists a fuller literature on 
school- and community-based evaluation programs and a few “how 
to” manuals on the basic elements of evaluation design (Kellogg 
Foundation, 1998). ^ 

A different set of challenges arises if a research partner is avail- 
able but has not done this type of evaluation before. Thus, there are 
several questions to ask the evaluator in advance of the assessment. 
These include: “What kind of evaluation experience do you have, 
i.e., what kinds of programs have you evaluated before?” “What 
methods have you used to gather evaluation data?” “How do you see 
the role of the evaluator on the project team?” “How well do you 
know our community or issues that may affect our target popula- 
tion?” “How will you handle data if findings are not what we expect- 
ed?” and “What experience do you have in providing feedback to 
policy makers and funders?” It is also reasonable to ask a potential 
evaluator if the program administrator can review copies of previ- 
ous evaluation reports. Budget and other program costs (e.g., indi- 
rect costs) should be discussed up front. The cost of conducting an 
evaluation can vary widely depending on the proposed design, 
staffing plan, and analysis plan. FinciUy, other program staff should 
meet with an evaluator. Conflicts and disagreements will sometimes 
occur. If the program staff and the evaluator do not get along well, 
the project will suffer greatly. 

Program administrators may also have pragmatic considera- 
tions related to conducting an evaluation. Will new forms or poli- 
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cies need to be developed to accommodate the evaluation compo- 
nent? Will new consent procedures and forms be required? How will 
the research issues affect the program’s ability to deliver services? 
Win program participants be willing to participate in an evaluation 
study? Can the program provide adequate access to its participants 
and appropriate data? Are there new legal issues related to evalua- 
tion staff access to data (e.g., legally sensitive information from 
police or courts) or regarding contact with program participants, 
especially minor youth? 

Challenges for the Evaluator 

One challenge for the evaluator can be the task of evaluating a 
vaguely defined intervention. If the program components are not 
obvious, it would be very difficult to evaluate their impact. 
Conversely, an evaluator may also he faced with evaluating a very 
complex program model; such models are increasingly common as 
best practice interventions. However, in a multi-system, multi-level 
program, it is extremely hard to determine which program compo- 
nent or combination of components causes a desired (or undesir- 
able) effect. The use of the logic model, which matches program ele- 
ments with specific evaluative questions, strategies, and methods, can 
be very helpful when evaluating complex service delivery models. 

As we stated earlier, randomization of program participants to 
intervention and nonintervention groups is the hallmark of rigor- 
ous, experimental research. Most evaluators are interested in the sci- 
entific merit of the data they are gathering. If a program is not 
designed to accommodate randomization, what are the options? 
We’ve talked about constructing a nonintervention comparison 
group, and of examining the program participants’ different levels 
of exposure to the program or intervention. However, from an eval- 
uator’s perspective, it may be necessary to commit significant 
resources to identifying, recruiting, and gathering information from 
subjects who are not benefiting from program participation. They 
are less accessible, less willing to provide information (especially 
without compensation), and less likely to participate over the long 
term. Much information already gathered can be utilized as valuable 
evaluative data. This includes information on services provided, 
units of treatment, number of contacts, participant demographics. 
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or tracking of program elements. Community-based organizations 
often gather data to meet requirements for service billing (e.g., 
Medicaid, mental health departments) and this information can be 
used as part of the evaluation. 

What should be the balance between quantitative versus quali- 
tative methods of gathering information? Ideally, decisions should 
be based on the method’s relevance to answering the specific evalu- 
ation questions. Further, it is necessary to consider accessibility to 
information and method (e.g., surveys vs. focus groups) and the 
resources available to the activity. Large-scale surveys can be expen- 
sive to conduct (e.g., gathering survey data from children, teachers, 
parents, and program staff) but may provide more information 
than a few targeted focus groups. 

How can a program administrator be confident that the data 
are accurate? It is important to make sure that the archival or 
administrative data used are accurate and reliable. They can be 
checked by doing some systematic independent data gathering. For 
example, if school data on discipline referrals Jo a principal are 
being used, a member of the research staff can randomly check with 
school staff or visit a school to determine whether the data are being 
gathered consistently and reliably. Alternatively, evaluation staff 
could develop a form for systematic information gathering across 
school sites for program evaluation purposes. Another option 
would be to commit resources to data gathering that are more 
directly under the evaluator’s control, rather than depending on 
others for information gathering. For example, the evaluation staff 
could spend dedicated time doing observations of students in a 
classroom rather than depend on teachers for completion of behav- 
ior surveys. Again, issues of cost, staff resources, time, and usefulness 
of information gathered need to be taken into account. 

Evaluators must always be aware of the implications of data gath- 
ering on human subject protocols and procedures, especially if they 
are gathering sensitive information from children on violent behav- 
ior, exposure to violence, delinquent behavior, or substance use. 

Many community-based programs and interventions in 
schools are impacted by significant subject mobility and attrition. In 
this respect, there are few true experimental designs in evaluation 
research. Most designs automatically become quasi-experimental in 
nature as soon as program participants drop out or the intervention 













group is nonequivalent to the comparison or control group. 
Creative, flexible evaluators should take mobility and attrition into 
account in their research design and data analysis plan. 

Neither program staff nor evaluators can control all the other 
factors or events occurring in the community, both positive and 
negative, which may affect program participants. Schools, for exam- 
ple, often have multiple intervention programs in place, some for 
substance abuse, others for gang resistance, and stiU others for gen- 
eral life skills development. At times the best the evaluator can do is 
document these events or programs. If program participant and 
control groups are similar, then we often assume that exposure to 
other events or programs in the community will be similar as well. 
Of course, this may not be the case in more unusual circumstances. 
When a multiple homicide occurs at a local school during the time 
a violence prevention program is in place, individual student expo- 
sure to that event, knowledge of the event, or relationship to victims 
may significantly impact students’ emotional status, behavior, etc. in 
unusual ways that would have to be taken into account as part of the 
program and evaluation. 

Evaluators also need to consider that normal developmental arti- 
facts may impact behavior outcomes in ways independent of program 
effects. Children become less prosocial as they mature, so declines in 
helpfulness may be normal developmentally and unrelated to program 
impact. Similarly, children in grades K-2 are often much more aggres- 
sive than children in grades 3-5. Having an understanding of these 
normal developmental phenomena will help the program and evalua- 
tor place in context and interpret observed outcomes. 

Sometimes, a program results in no effect, or even negative 
effects. Sometimes program participants do less well than similar 
groups of individuals who are not exposed to the intervention. 
Doing applied, community-based evaluation work is difficult and 
stressful, for both the program staff and the evaluation team. 
Sometimes researchers are trying to conduct relatively rigorous 
research in systems that are not set up to support such activity. The 
systems can be disorganized and strained themselves, with the pres- 
sure of working with high-need, high-risk children or families. 
Adding the strain of doing evaluation to this work can be over- 
whelming to staff. Goals of service delivery often compete with the 
goals of an evaluation team. Evaluation staff need ongoing support 
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and constant education about the process of gathering data in 
appUed settings. 

It is also essential to be sensitive to cultural issues in your com- 
munity. This could include a family s willingness to allow evaluators 
into their home and to ask sensitive questions. Sensitivity could also 
include paying attention to the construction of survey instruments 
and specific questions. 

Challenges for the Program 

It is important to be flexible in the program staff’s work with 
the evaluator. The challenges inherent in merging a program’s goals 
with evaluative goals are difficult enough to tackle. If either the eval- 
uator or program administrator is rigid in stance, then little of value 
will be accomplished. Of course, fi'om the program perspective, the 
fidelity and quality of the implementation of the program cannot be 
compromised just to meet the goals of information gathering for 
research or evaluative purposes. ,, 

It is also important to be willing to collect information from 
program participants in a systematic way so that data are reliable 
and meaningful. This may mean dedicating staff time for training 
by the evaluation staff in the use of instruments, procedures, and 
protocols for information gathering, and for imderstanding why 
they are collecting certain information and its role in service provi- 
sion and program support. 

A program may have to modify human subject protocols to 
include collection of evaluation specific or research data or to make 
data available to the evaluator that might not normally be accessible, 
to contact participants directly at a later time for potential follow-up 
(if the design calls for follow-up or focus groups). In some instances, 
procedures will need to be approved by an Institutional Review 
Board (IRB) for protection of human subjects. Most universities 
have IRB boards that would review and approve the protocol for 
data collection, management, and storage. The board will request 
changes if they are needed to bring the project into compliance with 
Federal regulations (e.g., obtaining active parental consent to gath- 
er data from minor children). It may be helpful if a program admin- 
istrator appears before the board with the evaluator to explain 
28 agency procedures and protocols. 
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An evaluator should not overwhelm the program services. An 
inexperienced evaluator may suggest assessment of everything and 
anything to try to ensure that nothing is missed. If aU contact time 
is spent gathering data, there may be no time left to provide services. 

Also, if participants are overwhelmed initially with data gathering, 
they may be less likely to agree to any follow-up. 

A program should recognize in its original design and in allo- 
cation of staff and other resources for the evaluation the time and 
effort that is needed to gather information, manage it, analyze it, and 
disseminate it effectively. Whenever possible, the evaluation staff’s 
need to access important, relevant information should be accom- 
modated. Sometimes this may mean modifying procedures or 
forms, or granting access to traditionally confidential information. 

- In such cases, the project wiU need to address changes in human 
subject protocols for data access and sharing. Much information 
already gathered can be utiUzed as valuable evaluative data. This 
includes information on services provided, units of treatment, 
number of contacts, participeint demographics, and tracking of pro- 
gram elements. Community-based organizations often gather data 
to meet requirements for service bihing (e.g., Medicaid, mental 
health departments) and, depending on confidentialiy issues, this 
information may be used as part of the evaluation. 

While the program probably will not get compensated for data 
gathering (not a billable service), collecting the information may be 
necessary to obtain ongoing support for the program and its staff. 

Political Considerations 

Evaluations of school and community-based programs are not 
conducted in a political vacuum. There are always program admin- 
istrators, policy makers, or funders who are waiting for outcome 
findings either to champion the program’s cause or to shift funding 
from the program to other initiatives. It would be foolish to ignore 
these issues in the conduct of the evaluation. From an evaluator’s 
perspective, it is important to understand that the program admin- 
istrator will not be able to answer all questions comprehensively, 
and that the evaluation will probably lead to new questions that can- 
not be answered in the short term. From the program perspective, it 
is important to understand that a commitment to a well-designed 29 
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evaluation will limit criticism of the findings, both positive and neg- 
ative, from individuals not directly associated with the program. 
From both the evaluator and program perspectives, it is important 
to try to consider the implications for policy from the outcomes, but 
not to be driven by those potential implications at all times. 

Sometimes an evaluation determines that a program does not 
have the intended, desired, or positive effects that were postulated. 
What should a program administrator do if negative effects are 
found? There is a great deal of pressure to find positive outcomes. 
This goes back to the original goals and objectives of the program 
and the evaluative questions asked. Not all findings will be exactly 
what was expected, and not all findings will be positive. The key is 
to learn from the program and have the data to drive future deci- 
sions about implementation or resources. 

One challenge is to deal with the pressure to make a system look 
effective. If a school- or community-based system or program agrees 
to data collection and evaluation because they are needed to support 
its work, what happens if the evaluator finds §ornething that the 
administrator thinks will make the program look bad? What if the 
evaluator finds that a disproportionate number of young people 
report using illicit substances or bringing weapons to school? What if 
program participants report dissatisfaction with the services they 
receive? It is important for the evaluator to come to agreement at the 
beginning with program staff about the propriety of information 
gathered, dissemination plans, and the use of data gathered for scien- 
tific publication. If a program is Federally or foundation funded, then 
the data gathered may be available by law for pubfic domain and 
review. By agreeing to receive the funds, the school or community 
program has agreed to these conditions and must abide by them. 
Potential negative findings can be minimized if the evaluator and pro- 
gram people work together from the beginning. 

Summary 

Conducting effective, efficient, and scientifically valid evalua- 
tions of community-based programs is difficult work, especially in 
light of the increasing complexity of multi-system, multi-level, com- 
prehensive interventions. The successful program will have program 
and evaluation staff who are willing to be flexible, creative, and per- 
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sistent. All will make mistakes, and sometimes the best thought-out 
decision will turn out to be the wrong one. Program and evaluation 
teams will not always agree on the best strategy, and sometimes goals 
and objectives will be in conflict. That is okay. The point is to learn 
from the mistakes and apply what is learned to the program. Being 
overly rigid will severely limit the program’s chances for success. 

It is key to remember that all elements of an evaluation impact 
on aU others. Doing the hard work early and well will significantly 
impact the information collected in the end, the ability to answer 
evaluation questions, and the utility of the program’s work for fun- 
ders and policy makers. More programs are expected to develop the 
infrastructure for long-term sustainability. Conducting a rigorous, 
reliable, and valid evaluation of the program is an essential compo- 
nent of its long-term sustainability and impact. 

Evaluating program effects is not just about assessing individual 
behavior change. There are many types of evaluation, and there is 
no gold standard for determinmg the scientific rnerit of the effort. 
In some cases, an evaluation will focus on systems-level impact, or 
on broader family or community outcomes, or on policy. These are 
all legitimate foci for an evaluation effort. Specific goals and out- 
comes should be determined by the program goals, objectives, and 
needs. The main question is no longer merely “Does it work?” 
Rather, the more appropriate question has become “Under what 
conditions and for whom does it work, and why?” 
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The Assessment of 
C oNFucT Resolution Programs 

Morton Deutsch 



In this essay, I present a framework for assessing conflict resolu- 
tion programs and indicate how education practitioners can do 
research which will enable them to reflect productively on their 
practice. There are many kinds of research, all of which have merit. 
They have different purposes and often require varying types of 
skills. There is a tendency among both researchers and practitioners 
to derogate research that does not satisfy their specific needs or does 
not require their particular expertise. This is a profound mistake, 
however, since both professional researchers and practitioners have 
much to contribute to the development of knowledge about conflict 
resolution programs. ' 

Types of Research 

/ 

Several types of research are relevant to the development and 
assessment of conflict resolution programs: basic research, devel- 
opmental research, field research, consurner research, action 
research, and research for self reflection upon an educator’s prac- 
tice. Although the insights and cooperation of the practitioner are 
valuable in aU forms of research, the skills of the professional 
researcher are particularly needed in basic, developmental, field, 
and consumer research. 

Basic Research 

Many unanswered questions regarding knowledge and practice 
in the field of conflict resolution still need to be answered. Among 
them are the following: 

• What are the reliable, valid, and reasonably precise ways of 
measuring the knowledge, attitudes, and skills- involved in 
constructive conflict resolution? 
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• What are the basic dimensions along which cultures vary in 
their response to, and management of, conflict? 

• What determines when a conflict is ripe for intervention or 
mediation? 

• What are the important similarities and differences in con- 
flict processes at the interpersonal, intergroup, and interna- 
tional levels? 

• What are the intervening psychological processes that lead to 
enduring and generalized change in managing conflict, and 
what are the psychological and social consequences of such 
change? 

• What differences exist among people with different types of 
personalities in their styles of conflict management? 

\ 

• What type of value system is implicit in the current practice 
of conflict resolution? 

These are only a few of the important questions that must be 
addressed if we are to have the kind of knowledge needed to make 
conflict constructive — whether it be in school, the family, industry, 
or community. These sorts of questions require systematic, extend- 
ed research which is directed at developing theory and the knowl- 
edge that would be useful in developing, as well as assessing, conflict 
resolution programs. 

Developmental Research 

Developmental research is concerned with helping to shape 
effective educational and training programs. Such research identi- 
fies the best ways of helping people acquire the knowledge, attitudes, 
and skills necessary for constructive conflict resolution. It answers 
such questions as; How should knowledge be taught (for example, 
the most effective type of curriculum)? For how long? The best 
strategies are apt to vary as a function of the age, educational level, 
cultural group, and personality of the children and adults involved. 
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There is a bidirectional link between development and basic 
research. To assess and compare the changes resulting from various 
educational and training programs, it is necessary to know what 
changes these programs were seeking to induce and also to develop 
valid and reliable measuring instruments and procedures for mea- 
suring the changes. 

There has been very little of the kind of research suggested here. 

Some evaluation research has indicated that conflict resolution 
training (CRT) is viewed as worthwhile by those using it; some has 
demonstrated that such training has positive effects on self-esteem 
and on reducing destructive forms of conflict (Deutsch, 1993; Lam, 

1989). We now need to go beyond demonstrating that CRT can be 
useful; we have to start studying what types of training are most 
effective and most efficient. 

Field Research 

\ 

Field research is needed to identify the features of political sys- 
tems, cultures, and organizations that facilitate or hinder effective 
CRT. For example, can CRT have desirable effects with inner-city 
high school students living and studying under adverse circum- 
stances? What kind of culture is most favorable to CRT training, and 
what kind makes it unfeasible or ineffective? Which levels in an 
organizational hierarchy must be knowledgeable and supportive of 
CRT for it to be effective? In schools, what type of CRT model 
should be employed: extracurricular activity, a specific course, in 
conflict resolution (CR), infusion into all school courses, use of con- 
structive controversy, or all of the above? Is cooperative learning a 
necessary precondition or a complement to CRT? Who should teach 
CR: a specialist in CRT, a teacher, a student, or a parent? What crite- 
ria should be employed in selecting CR trainers? 

Most of these questions have to be asked and answered in terms 
of the specific characteristics of the individual school, taking into 
account the resources, organization, personnel, student body, and 
social setting. Little if any research has been done on questions of 
this type because it is difficult and expensive to do such direct 
research. Experience surveys as a feasible alternative approach to 
such issues are discussed below. 35 
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Consumer Research 



It would be valuable to have periodic surveys of where CRT is 
taking place, who is being trained, what kind of qualifications the 
trainers have, and so on. Also, it would be good to know how CRT 
training is evaluated by its recipients, immediately after training and 
then one year later. In addition to studying individuals who have 
had CRT, it would be useful to assess what the market is for CRT 
among those who have not had it. 

Most of the research on CRT in schools and other organizations 
has been essentially study of “consumer satisfaction.” The research 
has usually involved studying the effects of CRT in a particular class- 
room, workshop, or school. Results are quite consistent in indicat- 
ing a considerable degree of approval among those exposed to CRT, 
whether in the role of administrator, teacher, student, or parent 
(Lim & Deutsch, 1997). 

Action Research 

Action research is a term originally employed by Kurt Lewin for 
research linked to social action. To be successful, it requires collabo- 
ration between the action personnel — the trainers, school staff, and 
practitioners- — and the research personnel. What the action person- 
nel do can be guided by feedback from the research concerning the 
effectiveness of their action. To study the process involved in suc- 
cessfully producing a change (or failing to do so) in a well-con- 
trolled, systematic manner, the researcher depends on the coopera- 
tion of the action personnel. Most research on CRT in the schools — 
no matter how it is otherwise labeled — is a form of action research. 

There are many potential sources of difficulty in this practi- 
tioner-researcher collaboration. It is time consuming and hence 
often burdensome and expensive to both the partners. Also, friction 
may occur because of their disparate goals and standards: one is 
concerned with improving existing services, the other with advanc- 
ing knowledge of a given phenomenon. The practitioner may well 
become impatient with the researcher’s attempt to have well-con- 
trolled independent variables (e.g., administration of the CRT pro- 
gram in essentially the same way in every classroom), and the intru- 
siveness involved in extensive measuring of the effects of the CRT 
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program on the student’s knowledge, attitudes, skills, and behavior. 
The researcher may become exasperated with the practitioner’s 
improvisation and reluctance to sacrifice time from other activities 
to achieve the research objectives. In addition, there is often much 
evaluation apprehension on both sides: the practitioners are con- 
cerned that, wittingly or unwittingly, they will be evaluated by the 
research findings; the researchers fear that their peers will view their 
research as not being sufficiently well controlled to have any merit. 



Research for Self-Reflection 

Although it is impossible for self-reflecting practitioners to 
assess whether a particular conflict resolution program is generally 
effective (e.g., effective when used by other teachers, or with differ- 
ent students, or in other school cultures), they can find out what 
effects the CRT program that they are using has on their own stu- 
dents. To do so, practitioners will need to ask such questions as the 
following: 

• What are my objectives in using this program? Do I want it to 
help in establishing a peaceful, orderly classroom? To help 
students manage their conflicts more constructively — in 
school? At home? In everyday life? To improve their grades? 

• How will I know if my objectives are achieved? What kinds of 
information will I need to collect and how will I collect it? Is 
there a decrease in negatives (such as fights, bullying, victim- 
ization, verbal abuse, hurt feelings, antagonisms, discrimina- 
tion, and disorder) and an increase in positives (such as a 
willingness to face problems openly; maintaining respect for 
the other during conflict; working cooperatively to resolve 
conflicts; listening to and communicating with the other; 
confidence in ability to deal with problems; or better relations 
with peers, teachers, and family members; as well as increased 
harmony and order in the classroom)? 

• How will I use my observations, self-reports by students 
(obtained through interviews, questionnaires', or diaries), and 
reports by others about the student (e.g., other students, other 
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teachers, parents, staff) to provide me with the information I 
need to assess what changes, if any, have resulted from CRT? 

• If the CRT program appears to have achieved its objectives, is 
it because there has been a real change in the students (e.g., 
they have decreased their negatives and increased their posi- 
tives through exposure to CRT) or because the students were 
good conflict resolvers prior to training? If there has been a 
desirable change, is it due to CRT or to some other factor, 
such as the increased maturity (age) of the student, or to the 
introduction of some other change in the curriculum, school, 
or neighborhood? 

• If CRT does not achieve my objectives, how can I determine 
the reasons for the failure? Are other teachers having similar 
CRT results or are some having better results? If poor results 
are common, are they due to identifiable inadequacies in the 
CRT program; to inadequate training or su'pport for imple- 
menting the program successfully; or to a countering influ- 
ence in the school, families, or neighborhood? If poor results 
are not common, are there important differences between the 
way I and my more successful colleagues implement the CRT 
program, or are the differences mainly in the nature of the 
student groups being taught? 

Self-reflecting teachers and CRT trainers will be aware of the 
natural tendency to think that what they do has desirable effects, 
and will be appropriately skeptical and aware that they need specif- 
ic evidence that whatever positive effects have occurred are, in fact, 
due to their efforts. 

Research Strategies 

Many factors make it very difficult to do research on the ques- 
tions outlined in the previous sections, particularly the kind of ide- 
alized research that most researchers would prefer to do. For exam- 
ple, it is rarely possible to assign students (or teachers, or adminis- 
trators) randomly to be trained (or not trained) by randomly 
3g assigned expert trainers employing randomly assigned training pro- 
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cedures. Even if this were possible in a particular school district, 
there would be the possibility that the uniqueness of the district had 
a significant impact on the effectiveness of training; no single dis- 
trict can be considered an adequate sample of all or other school 
districts. Employment of an adequate sample (which is necessary 
for appropriate statistical analysis) is very costly and probably nei- 
ther financially nor administratively feasible. 

Given this reality, what kind of research can be done that is 
worth doing? Here I outline several mutually supportive research 
strategies of potential value. 

EXPERIMENtAL AND QUASI-EXPERIMENTAL RESEARCH 

Experimental research involves small-scale studies that can be 
conducted in research laboratories, experimental classrooms, or 
experimental workshops. It is most suitable for questions related to 
basic or developmental research, questions specific to what is to be 
investigated. Thus, experimental research would be appropriate for 
testing the hypothesis that role reversal does not facilitate construc- 
tive conflict resolution when the conflict is about values (such as 
euthanasia) but is constructive when interests are in conflict. 
Similarly, it would be appropriate for examining the relative effec- 
tiveness of two different methods of training in such conflict reso- 
lution skills as perspective taking and reffaming. 

This kind of research is most productive if the hypothesis or 
question being investigated is well-grounded in theory or in a sys- 
tematic set of ideas rather than ad hoc. If well-grounded, such 
research has implications for the set of ideas within which it is 
grounded and thus has more general implications than does testing 
an ad hoc hypothesis. It is necessary to be aware, however, that in this 
type (as well as in aU other types) of hypothesis-driven research, a 
hypothesis may not be supported — even if it is valid — because 
implementation of the causal variables (such as the training meth- 
ods), measurement of their effects, or the research design may be 
faulty. Generally, it is easier to obtain nonsignificant results than to 
find support for a hypothesis. Thus, practitioners have good reason 
to be concerned about the possibility that such research may make 
their efforts appear insignificant even though their work is having 
important positive effects. 
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The logic involved in true experiments assumes that complete 
randomization has occurred for all other variables except the causal 
variables being studied. But human beings have life histories, per- 
sonalities, values, and attitudes prior to their participation in a con- 
flict resolution workshop or experiment. What they bring to CRT 
from their prior experience may not only influence the effectiveness 
of the training being studied but also be reflected directly in the 
measurement of its effects. Thus, an authoritarian, antidemocratic, 
alienated member of the Aryan Nation Militia Group may not only 
be unresponsive to CRT, but also independently of this, may score 
poorly on such measures of the effectiveness of CRT as ethnocen- 
trism, alienation, authoritarianism, and control of violence because 
of his or her initial attitudes. Such people are also less likely to par- 
ticipate in CRT and be responsive to it than democratic, nonviolent, 
and nonalienated people. 

With appropriate “before” measures and correlational statistics, 
it is possible to control for much (but far from all) of the influences 
of initial differences in attitudes on the “after” rheasures. In other 
words, a quasi-experiment that has some resemblance to a true 
experiment can be created despite the prior histories of the people 
who are being studied. 

Causal Modeling 

Correlations, by themselves, do not readily permit causal infer- 
ence. If a negative correlation between amount of exposure to CRT 
and authoritarianism is found, its cause may be the fact that those 
who are authoritarian are less apt to expose themselves to CRT, as I 
have suggested, or that those who have been exposed to CRT 
become less authoritarian, or the causal arrow may point in both 
directions. It is impossible to tell from a simple correlation. 
However, methods of statistical analysis developed during the past 
several decades (and still being refined) enable appraisal with con- 
siderable precision of how well a pattern of correlations within a set 
of data fits an a priori causal model. Although causal modeling and 
experimental research are mutually supportive combinations, causal 
modeling can be employed even if an approximation to an experi- 
mental design cannot be achieved. This is likely to, be the case in 
most field studies. 
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Consider, for example, a study we completed several years ago 
on the effects of training in cooperative learning and conflict reso- 
lution on students in an alternative high school (Deutsch, 1993; 
Zhang, 1994). Prior theoretical analysis (Deutsch, 1949; 1973; 
Johnson & Johnson, 1989), as well as much experimental and quasi- 
experimental research (see Johnson & Johnson, 1989, for a compre- 
hensive review), suggested what effects such a training could have 
and also suggested the causal process that might lead to these effects. 
Limited resources made it impossible to do the sort of extensive 
study of many schools required for an experimental or quasi-exper- 
imental study, or to employ the statistical analysis appropriate to an 
experiment. So we created a causal model that, in essence, assumed 
training in cooperative learning and/or conflict resolution would 
improve the social skills of a student. This, in turn, would produce 
an improved social environment for the student (as reflected in 
greater social support as well as less victimization from others), 
which would lead to higher self-esteem and more sense of personal 
control over one’s fate. The increased sense of control would 
enhance academic achievement. It was also assumed that improve- 
ment in the student’s social environment and self-esteem would 
lead to an increased positive sense of well being as well as decreased 
anxiety and depression. The causal model indicated what we had to 
measure. Prudence suggested that we also measure many other fac- 
tors that potentially might affect the variables on which the causal 
model focused. 

The results of the study were consistent with our causal model. 
Even though the study was quite limited in scope — having been 
conducted in only one alternative high school — the results have 
some general significance. They are consistent with prior theory and 
also with prior research conducted in very different and much more 
favorable social contexts. The set of ideas underlying the research 
appears to be applicable to students in the difficult, harsh environ- 
ment of an inner-city school as well as students in well-supported, 
upper-middle class elementary and high schools. 

Survey Research 

This form of research is widely used in market research; pre- 
election polling; opinion research; research on the occurrence of 




crime; collection of economic data on unemployment, inflation, 
sales of houses; and so on. A well-developed methodology exists 
concerning sampling, questionnaire construction, interviewing, and 
statistical analysis. Unfortunately, little survey research has taken 
place in the field of CRT. Some of the questions that could be 
answered by survey research have been discussed above, in the 
Consumer Research section. 

Experience Surveys 

Experience surveys involve intensive in-depth interviews with a 
sample of people, individually or in small focus groups, who are 
considered to be experts in their field. Often the purpose of such 
surveys is to obtain insight into the important questions needing 
answers through the experts’ identification of important gaps in 
knowledge or through opposing views among the experts on a par- 
ticular topic. In addition, interviewing experts prior to embarking 
on a research study generally improves the researcher’s practical 
knowledge of the context within which the research is conducted 
and applied, thus helping the researcher avoid the minefields and 
blunders caused by naivete. 

Most important, experts have a fund of knowledge, based on 
their deep immersion in the field that may suggest useful, practical 
answers to questions that would be difficult or unfeasible to answer 
through other forms of research. Many of the questions mentioned 
earlier in the Field Research section are of this nature. Of course, the 
researcher’s confidence in the answers of the experts is eventually 
affected by how much agreement or disagreement is manifested. 

There are several steps involved in an experience survey. The first 
is to identify the type of eq^ert respondent desired for the survey. For 
example, with respect to CRT in schools, it might be useful to survey 
practitioners (the trainers of trainees), teachers who have been 
trained, students, or administrators of schools in which CRT has been 
provided. The second step is to contact several experts of the chosen 
type and have them nominate other experts, who in turn nominate 
other experts. After several rounds of such nominations, a group of 
nominees usually emerges as being widely viewed as experts. The 
third step is to develop an interview schedule. This typically entails 
formulating a preliminary schedule that is tried out and modified as 




a result of interviews with a half dozen or so of the experts individu- 
ally or in groups. The revised schedule is formulated so as to ask the 
experts all of the questions whose answers are sought, while leaving 
them the opportunity to raise issues and answer the questions in a 
way that was not anticipated by the researcher. 

What TO Measure 

The objective of most CRT programs is to affect the knowledge, 
attitudes and emotions, skills, and behaviors of the participants so 
that when they are involved in a conflict they are more likely to ini- 
tiate and develop a constructive process of conflict resolution. Thus, 
assessment of a CRT program may measure the effects of the pro- 
gram on students’ knowledge of conflict processes; on their orienta- 
tion, attitudes, and emotions toward a conflict; on the component 
skflls involved in constructive conflict resolution; and on behavior 
in diverse conflict situations. Below, some of the variables that might 
be measured are briefly oudined. \ 

Knowledge 

r 

This assessment covers the students’ acquisition of knowledge 
in the following areas: (1) the typical steps involved in both con- 
structive conflict resolution and mediation processes and (2) basic 
concepts, such as reframing, cooperation, corhpetition, active listen- 
ing, responsive cooperation, mutual problem solving, the distinc- 
tion between needs and positions, taking the perspective of the 
other and role reversal, ethnocentrism and cultural differences, mis- 
perceptions, and “hot buttons” in self and other. 

Orientation, Attitudes, and Emotions 

This assessment measures whether the students have acquired 
the orientation, attitudes, and emotional responses to conflict which 
facilitate constructive rather that destructive conflict resolution. It 
helps determine whether the students have developed a cooperative 
(win-win) rather that a competitive (win-lose) orientation to con- 
flict, with positive (hopeful, interesting, creative) rather than nega- 
tive (fearful, anxious, avoidant, antagonistic) expectations and feel- 
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ings about conflict resolution processes and outcomes. Assessment 
might also cover the effects of the CRT program on social attitudes 
such as alienation, trust, suspicion, ethnocentrism, authoritarian- 
ism, and power-orientation. 

Skills 

The ultimate test of the effectiveness of a CRT program is 
whether the participants have acquired the skills that are needed for 
constructive conflict resolution and whether they use them in the 
conflicts they experience in their daily lives. The following skills are 
the focus of many CRT programs. 

• Ability to “break the ice,” reduce tension, find common 
ground between oneself and the other, and establish a friend- 
ly working relationship. 

\ 

• Ability to reframe the conflict so that it is viewed as a mutual 
problem to be resolved cooperatively. 

• Ability to communicate effectively and responsively with the 
other, and to listen actively to the other, which involves under- 
standing the meaning and emotions involved in what the other 
is communicating and checking the correctness of each partic- 
ipant’s understanding of the other’s communication. 

• Ability to engage in perspective taking and role reversal, 
which involve empathic understanding of the other’s situa- 
tion and social cultural context as well as the other’s position 
and underlying needs. 

• Ability to differentiate between “position” and “needs” and to 
identify the underlying needs in oneself and the other. 

• Ability to problem solve, including being able to diagnose the 
nature of the conflict, creatively generate potential solutions 
to it, and evaluate the alternative solutions in terms of their 
feasibility, durability, desirability, and fairness. 




• Ability to support, encourage, and enhance the other. 

.* Ability to control personal impulses (e.g., resist being overly 
angry, over-conciliatory, and ethnocentric; overreacting when 
one’s “hot buttons” are pushed; and being defensive). 

• Ability to respond constructively to the other s emotional out- 
burst, attacks, ethnocentrism, “hot buttons,” and defensiveness. 

• Ability to deal with “dirty tricks,” “deception,” and the other’s 
unwillingness to cooperate. 

Behavior in Different SrruAnoNS 

Individuals are sometimes more able to employ their knowl- 
edge, attitudes, and skills to manage their conflicts constructively in 
some types of situations than in others, with some types of people 
and not others, and about some types of issues and not others. It is 
useful to know what types of situations, people, and issues are prob- 
lematic for a student. It is impossible here to set forth a comprehen- 
sive classification of types of situations, people, and issues, but some 
distinctions that are relevant to the ease of resolving a conflict are 
identified here: 

• It seems likely that the knowledge acquired in CRT can read- 
ily be transferred to situations which are characterized by 
strong norms of cooperation and also values such as reci- 
procity in fairness, human equality, shared community, 
recognition of personal fallibility, and nonviolence (see 
Deutsch & Coleman, 2000, Chapter 2). It can be expected that 
if a situation is characterized by radically different norms and 
values a transfer would be difficult. Thus, if the school culture 
is authoritarian and competitive, it may be difficult to make 
the transfer from the more cooperative, egalitarian culture in 
a CRT classroom. Similarly, the likelihood of the transfer 
from CRT in the classroom to conflicts in the family, work, or 
community settings will be affected by the norms and values 
in these different settings. 
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• If the other approaches conflict with the norrns and values 
underlying constructive conflict resolution, transfer is like- 
ly — providing that an individual’s relations with the other 
does not lead to abandonment of these norms and values. 
There has been no systematic research in this area, but it 
seems likely that transfer is more apt to take place in conflict 
with peers than with superiors or subordinates, and with oth- 
ers whose personalities are compatible with the individual’s 
own rather that incompatible. 

• Issues that threaten personal or important group identities, 
esteem, security, or survival are difficult to resolve construc- 
tively, as are issues that have a long history of contentious, 
unresolved dispute. In addition, conflicts over basic values, 
relative power, relative status, or possession of limited 
resources vital to security, esteem, identity, or power are diffi- 
cult to resolve constructively unless the parties involved in the 
conflict are highly skilled and are strongly^ committed to the 
norms and values underlying constructive conflict resolution. 



How TO Measure 

There are a number of different forms of measurement; obser- 
vation, interviews, questionnaires, diaries, and records. 

• Observation of actual behaviors in real conflict situations is 
probably the most persuasive form of data collection, but also 
the most difficult and costly. However, teachers who have 
continuing contact with their students may be in a good posi- 
tion to observe changes in behaviors in their classrooms. 
Because observation of real conflicts is difficult, observation 
of “simulated conflicts” is commonly used as a substitute. 
Here, the teacher assigns students to take a given role in a sit- 
uation that simulates a real conflict (e.g., a conflict between 
two friends, a parent and child, a student and teacher) and 
observes their behavior. 

• Interviews and questionnaires involve obtaining reports from 
the students about their own knowledge, attitudes, skills, and 
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behavior in conflict situations. An alternative is obtaining 
such reports from others in a position to observe the student’s 
behavior (e.g., fellow students, neighbors, friends, parents, 
teachers, supervisors, and subordinates). Interviews with 
individuals or focus groups are a much more flexible way of 
obtaining useful information than questionnaires, but also 
much more costly and time consuming. 

• Review of diaries in which the student records daily experi- 
ences with conflict can be of considerable value, especially if 
students are given a framework or set of questions to use as a 
guide. 

• Review of data found in records of various sorts can also be 
useful to determine whether there is a decrease in student 
violence, delinquency, vandalism, disciplinary cases, absen- 
teeism or truancy, health complaints, depression, and neurot- 
ic symptoms; or an improvement in school grades, voluntary 
actions to help the class or school, cooperative activities 
among teachers, and so forth. 

Resources 

Several types of publications may be useful for those who want to 
be self-reflecting practitioners; those dealing with learning through 
reflection, those focusing on research methods, and those describing 
research that has been conducted and research methods that have 
been utilized in evaluating conflict resolution training. Some valuable 
references in each of these three areas are presented below. 

Learning Through Reflection 

Marsick and Sarquet (2000) have written an excellent chapter 
on this topic and their list of references contains additional useful 
reading, including Marsick and Watkins (1999), Mezirow (1991), 
and Schon (1987). 
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Research Methods 



The classic textbook on research methods by Judd, Smith, and 
Kidder (1991) contains excellent chapters on all aspects of research, 
including interviewing, questionnaire construction, observation 
methods, as well as other forms of data collection. The monograph by 
Robinson, Shaver, and Wrightsman (1991) contains descriptions and 
examples of a great number of widely employed rtleasures of person- 
ality and social attitudes, some of which may be influenced by CRT. 

Research Studies 

The authors of the various chapters in The Handbook of Conflict 
Resolution (Deutsch & Coleman, 2000) make reference to a wide 
variety of studies that are relevant to CRT, and that contain specific 
measures for assessing the effects of CRT. A section of Chapter 27 
also provides a summary of research findings on CRT. Bodine and 
Crawford (1998), in their own chapter, “Research Findings on What 
Works,” in their collection also present a relevant summary. See Lam 
(1989) for a review of the impact of conflict resolution in schools, 
and Elliott, Hamburg, and Williams (1998) for a review of research 
on programs to prevent school violence. 

In addition, there are a number of recent, well-designed studies 
which have employed measurement instruments that are appropri- 
ate for students at different ages. For preschoolers, Sandy and 
Boardman (in press) have described various ingenious measuring 
instruments that they have employed in The Peaceful Kids Conflict 
Resolution Program, as well as the measures they used with parents 
and the day care staff. (Contact Dr. S.V. Sandy at the International 
Center for Cooperation and Conflict Resolution, Box 53, Teachers 
College, Columbia University, New York, NY 10027.) 

Johnson and Johnson (2000) have described a series of studies 
with students ranging from kindergarten through ninth grade in 
their Teaching Students to Be Peacemakers: Results of Twelve Years of 
Research. Their measures are briefly described in this paper. 
(Contact Professor David W. Johnson at the University of 
Minnesota, 60 Peik Hall, Minneapolis, Minnesota, 55455.) 

Aber, Brown, and Heinrich (1999) have conducted systematic 
4 g research, using very interesting measures, with children in elemen- 
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tary grades. (Contact Professor John L. Aber at the National Center 
For Children in Poverty, Joseph L. Mailman School of Public Health 
of Columbia University, New York, NY, 10032.) 

Jones (Jones, 1997; Jones & Kmitta, 2000) has done extensive 
research, using a variety of well-developed measures, on students in 
elementary, middle, and high schools. (Contact Professor Tricia S. 

Jones at the Department of Communication Sciences, Temple 
University, Philadelphia, PA, 19122.) 

Coleman and his work group (Coleman & Lim, in press) have 
constructed systematic questionnaires for evaluating CRT for use 
with adults who were trained and other instruments for use with 
people who know the trainees well, and can report on changes in the 
behavior of the trainees. (Contact Professor Peter Coleman at the 
International Center for Cooperation and Conflict Resolution, 
address above.) 

Summary AND CoNcxusiONS 

\ 

In this paper, I have presented a framework for thinking about 
the assessment of conflict resolution programs. I have also briefly 
discussed different types of research and research strategies, as well 
as ways of measuring the kinds of effects that might be achieved 
from such programs. Although self- reflective practitioners will often 
have neither the resources nor the advanced training in research 
methods to do much of the research outlined in this paper, person- 
al self-reflection can become deepened and more systematic by care- 
ful consideration of the issues presented here. 
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Evaluating School and 
Community-Based 
Violence Prevention 
Resource Guides: 

What to Look for, What to Do 

Daniel J. Flannery and Megan Seaman 



Introduction 



Despite recent downturns in national rates of juvenile violence, 
many children and school staff continue to report that they do not 
feel very safe at school. In fact, while the overall rate of violence at 
school has decreased in recent years, the number of multiple homi- 
cide incidents at school has increased (Kaufrnan et^al., 1998). These 
rare incidents have received a great deal of attention in the media, 
usually immediate and intense. This has contributed to the percep- 
tion of children and adults that schools are not as safe as they used 
to be. In many communities, legislators and school administrators, 
law enforcement and mental health professionals have begun to 
examine their school safety policies and initiatives to prevent or 
reduce violence (Decker, 2000). 

This renewed focus on schools comes at a time when the area of 
violence prevention in general and school violence in particular has 
changed tremendously. The knowledge base for what is effective sci- 
ence-based practice has grown significantly, as has the sheer number 
of intervention strategies now available to schools and community- 
based practitioners. Along with the implementation of more pro- 
gramming to intensify violence prevention efforts, there is increased 
emphasis on evaluating the impact of those programs on student 
behavior, school climate, and policies related to violence prevention. 

A good assortment of resource guides has been developed 
recently that focus specifically on violence prevention programs. 
This paper provides information about the availability of these 
guides and reviews in more detail a subset of them with respect to 
their utility for school or community-based professionals who are 
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developing a violence prevention strategy. 

Our review is not meant to be exhaustive or comprehensive. 
Rather, the selection of guides considered in more detail is illustra- 
tive of the different kinds of resources available. While our decisions 
are largely based on the areas or content covered, we also wanted to 
review a variety of different types of manuals. For example, some 
guides focus on school-based programs specific^y, whereas others 
cover community-based programs. Some resource guides are geared 
toward mental health professionals or school counselors. Some 
focus on targeted youth programs, others on universal prevention 
or public health programs. Because of our primary concern with 
school-based violence prevention strategies, we do not include a 
review of guides specific to other areas of prevention such as sub- 
stance abuse or family violence. 

This essay is organized into several sections. First, we lay out our 
methodology for identifying the 33 resource guides reviewed. Next we 
discuss the two tables that provide information on the guides, which 
are presented at the end of the paper. Table 1 describes each guide’s 
primary subject area (including coverage of evaluation as a topic) and 
target audience, and provides information on acquiring the guide. 
Table 2 summarizes each guide’s coverage of particular content. In 
some instances we were able to indicate whether a topic was covered 
in depth or just mentioned as a necessary component (e.g., evalua- 
tion, staff training). We then present a more detailed review of seven 
guides selected on the basis of the number of content areas covered 
and the general focus of the guide. Finally, we offer some summative 
comments about the strengths and weaknesses of resource guides for 
school-based violence prevention programs. 

MEraODOLOGY 

Our initial strategy in searching the literature was very broad. 
With a general focus on youth violence prevention and a more spe- 
cific interest in school-based violence prevention strategies, we con- 
ducted a systematic search of the Internet. Guides that covered 
school or community-based violence prevention strategies, whether 
they were universal or targeted to at-risk youth, were included. We 
also included resource guides that discussed risk and protective fac- 
tors related to youth violence and guides that reviewed safe school 




practice and policies. We searched governmental agencies, research 
institutes, private foundations, and academic institutions. We did 
not strictly confine our search to resource guides and manuals, 
because we also came across several books, research briefs, and spe- 
cial editions of academic journals that focused specifically on 
school-based violence prevention. Some of these resources are 
included in our tables, and others are referenced as appropriate in 
our narrative. 

We were also interested in reviewing resource guides for the 
non-researcher or for the practitioner who may not have ready 
access to an academic institution. We found guides that targeted a 
variety of audiences, from school staff and administrators to law 
enforcement personnel to community-based organizations. Our 
review therefore contains guides that were developed specifically for 
each of these targeted groups of professionals. 

Finally, we restricted our search to more recent guides, and 
decided to include those produced since 1995. In fact, we excluded 
very few resources because they were developed earlier; most of the 
guides we located were written in the past three years. This fact illus- 
trates the growing attention to the issue of violence prevention and 
the accumulating knowledge base for identifying, implementing, 
and evaluating best practices. 

Our first task was to provide an overview of the focus and avail- 
ability of each of the guides reviewed (Table 1 ) . Our second task was 
to provide a more substantive review of the coritent of each of . the 
resource guides contained in Table 1. The content areas selected for 
inclusion reflect the topics most typically covered by the resource 
guides reviewed. Specifically, we considered whether the guide pro- 
vided current data on the prevalence or incidence of school violence 
to provide context for the kinds of programs reviewed. “Programs” 
and “Curriculum” refer to whether the guide systematically 
reviewed specific school- or community-based violence prevention 
programs or specific curricula instead of offering general informa- 
tion about strategies or issues. We also note in Table 2 whether a 
resource guide discussed the importance of a school’s forming part- 
nerships with other community-based agencies in their violence 
prevention planning and the steps to take in building those partner- 
ships. Several of the substantive categories in Table 2 are self- 
explanatory, as in whether the guide discussed risk or protective fac- 53 
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tors, whether it covered specific programs or services to targeted, 
high-risk youth (versus primary prevention programs), and 
whether it included general safe school planning as part of a com- 
prehensive violence prevention initiative. 

We looked for coverage of several content areas specifically 
because of their inconsistent coverage across manuals. For example, 
we tried to identify whether resource guides covered program eval- 
uation as a separate topic (indicated with two check marks on Table 
2) or just indicated in a more cursory way that evaluation is a nec- 
essary component of a program’s effort. Similarly, we noted whether 
programs were presented in a developmentally appropriate context, 
whether they contained any discussion of the role of staff training, 
whether they covered identification of and support for at-risk 
youth, and whether they contained some discussion of the chal- 
lenges or barriers to implementing an effective violence prevention 
strategy. 

Based on our review of substantive areas covered, we then 
selected seven resource guides to consider in further detail (shaded 
in the tables). They were chosen because they covered the greatest 
number of topics and were examples of different types of guides, 
each with a different focus or target audience. Some guides covered 
most of the areas considered important, but did not go into much 
detail on the specific methods of undertaking the tasks related to the 
specific area. Some guides presented detailed information on the 
important areas, but focused on only one strategy of violence pre- 
vention. In review of aU of the included resources, it was apparent 
that not one single guide or manual had all the answers to imple- 
mentation of a school- or community-based violence prevention 
program and/or curriculum. 

Criteria for Review of Resource Guides 

For our more in-depth review of selected resource guides, we 
chose categories that we thought were most relevant to the non- 
research oriented school or to the community-based professional 
interested in implementing violence prevention programs in their 
particular setting. Since guides varied fairly significantly on sub- 
stantive areas of coverage, we felt it important to have a sense of 
what a guide did well and what it was not intended to do. Our 
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review in this essay of these substantive areas was meant to comple- 
ment and in some cases expand on the content areas covered in 
Table 2. 

The first consideration was the intended or target audience of a 
resource guide. While it could be argued that effective violence pre- 
vention should be multi-disciplinary and information provided for 
one group would be helpful to other professionals, the reality is that 
most guides are written with a specific group of professionals, types 
of programs, or contexts in mind. Thus, a school administrator 
interested in broad-based strategies for school-based violence pre- 
vention may not find a guide developed specifically for counselors 
with emotionally disturbed youth the most helpful. 

The second criterion for review was the breadth or scope of infor- 
mation covered by the resource guide. Each guide tended to cover one 
or a few areas in more detail than others. Few guides gave significant 
breadth of coverage to every potential topic (e.g., from policy to staff 
training and implementation guidelines to evaluation work). Brevity 
of coverage may be a strength if the guide were n\eant solely as a 
resource or reference to other material, but in some instances a guide 
may be limited by the brevity of coverage of an important topic. 

The third area considered was the organization and readability of 
the resource guide. This was a rather subjective assessment, but an 
important consideration for the professional not familiar with the 
area of violence prevention. Fourth, we considered whether recom- 
mendations for best practice or for developing effective strategies 
were theory- or information-based. Too many programs are available 
that have not been evaluated or developed based on what we know. 

Fifth, we considered whether the guide responded to the need 
for a multi-risk, multi-system, comprehensive (but flexible) strategy, 
since recent research on effective violence prevention strategies has 
consistently pointed to it as important. Singular, time- and risk-lim- 
ited approaches are generally not effective strategies in the arena of 
violence prevention. We also considered whether a resource guide 
discussed the role and importance of staff training. Preventive inter- 
ventions that are not implemented by well-trained, highly qualified 
staff are not effective. Fidelity of program implementation over time 
is essential for effectiveness and for long-term sustainability. A well- 
trained, dedicated staff is a significant resource that provides the 
capacity for accurate and effective implementation. 
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Seventh, we examined whether a guide discussed the role of 
evaluation in program planning and implementation for best prac- 
tice. In some cases, this meant the guide noted the importance of 
conducting evaluation of program effectiveness. In other cases, it 
meant the guide discussed the evaluation data related to the recom- 
mendations for programs contained in that resource guide. We have 
found generally that evaluation is often a recommended or even a 
required component for implementing school- and community- 
based violence prevention initiatives, but there are few resources 
available to the non-researcher on how to do evaluation pragmati- 
cally. Rarely did inclusion of evaluation as a topic outline specific 
strategies for conducting evaluations of violence prevention pro- 
grams or initiatives. 

Finally, we considered whether the resource guide discussed 
barriers or challenges to implemention of the programs or strategies 
recommended and whether it offered additional resources or infor- 
mation to support the efforts recommended. No program or best 
practice can “come in a box” ready to be dropped into a school and 
be expected to produce immediate, effective results. Every school 
and community will have its own unique environments, circum- 
stances and challenges that will have to be addressed for any violence 
prevention initiative to be successful. 

Review of Selected Resource Guides 

(1) Annual Report on School Safety, 1999, U.S. Departments of 

Education and Justice, 1999 (#1 in Tables) 

The Annual Report targets a broad-based audience, including 
school staff and administrators, mental health professionals, and 
juvenile justice and public health workers. It is specific to school set- 
tings. This resource guide is well organized and easy to read, with eas- 
ily understood graphics and charts. It provides an in-depth assess- 
ment of factors specific to the prediction and prevention of violence, 
and to preparation for dealing with violence that occurs in schools. 

One of the nice features of the Report is its highlight of relevant 
statistics and recent studies on the topic of school violence. For 
example, it points out that most school- related injuries are non-fatal 
and unintentional, not due to violence. The Report also summarizes 
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data showing that theft is the most common form of crime at 
school, and that more violent crime victimization for 12- to 18- 
year-olds occurs away from school than at school. In general, 
schools are relatively safe places with risk for homicide or suicide at 
school or at a school-related event less than one percent of all homi- 
cides and suicides. The Annual Report uses data to highlight the fact 
that middle school children and teachers suffer the highest rates of 
victimization from violence. In general, the Report does a solid job 
of explaining the nature and scope of school violence (from both a 
domestic and comparative international perspective) and of intro- 
ducing the rationale for selected model programs in the context of 
what we know about school-associated risk for violence victimiza- 
tion and perpetration. The Annual Report vrd& one of the few guides 
to discuss specifically the issue of hate crimes and harassment as ele- 
ments of school violence. 

The remainder of the Report summarizes information on model 
prevention programs in the areas of violence, substance use, and 
problem behavior, updating information from the 1998 Annual 
Report. It highlights specific grades (developmental appropriateness) 
and the target audience for the intervention, and provides informa- 
tion on ordering program materials. There are also several specific 
schools highlighted with brief summaries of their experiences in 
implementing best practice programs. These profiles could be par- 
ticularly helpful to school personnel seeking examples of implemen- 
tation success and resources to contact to discuss their challenges 
with implementation. The report lacks any direct or specific discus- 
sion, however, of potential barriers to implementation or guidance 
on how to overcome challenges associated with implementation. 

One of the few weaknesses of the Annual Report is the lack of 
any substantive discussion of the role of staff identification, training, 
and support. Further, there is the implied but not formally stated 
implication that model programs are multi-system, comprehensive, 
and long term. There is a nice compendium of resources, including 
other organizations, web sites, online Federal documents and 
Federal resources for individuals interested in doing school-based 
violence prevention work. 
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(2) Best Practices of Youth Violence Prevention: A Sourcebook for 
Community Action, Centers for Disease Control and Prevention, 
2000 (#3 in Tables) 

This resource guide offers a thorough, easy-to-read, and well- 
organized discussion of the elements required to implement an 
effective, best practice violence prevention strategy. While the 
Centers for Disease Control and Prevention (CDC) is best known 
for its public health oriented work in violence prevention, its 
Sourcebook would appeal to a wide audience in schools and com- 
munities. It was prepared with the help of a national panel of study 
group members from diverse backgrounds and disciplines. This 
diversity shows in the final product. 

The SourcebooKs first chapter is a well-organized overview of 
tasks related to the planning, implementation, and evaluation of 
violence prevention interventions. The “cookbook” approach to 
describing the problem, selecting participants and appropriate set- 
tings, setting goals and objectives, etc. would appeal to the novice 
evaluator or community-based professional faced with the daunting 
task of developing an initiative from “scratch.” Its expanded focus on 
staff selection, training, and support, and the need to garner local 
resources and support from the community, is particularly unique 
to these types of guides and very helpful. 

The CDC’s Sourcebook has a strong research and theoretical 
base for its focus on the specific “best practices” of parent-family 
based strategies, home visitation, social-cognitive strategies, and 
mentoring programs. The structure of the overview chapter is 
maintained in each of the sections on selected strategies, which is 
helpful to the reader who may skip that chapter to focus on a par- 
ticular type of intervention. Each of the strategies in the Sourcebook 
is covered in-depth, with resources provided for additional infor- 
mation about each one, including program summaries and con- 
tacts. There is considerable space given to a listing of the Healthy 
Families American Research Network participants at the end of the 
section on home visitation strategies. 

A strength of the Sourcebook is that staff selection, training, and 
support issues particular to each type of intervention strategy are 
included. For example, staffing issues are very different for imple- 
menting a home visitation program versus a mentoring program. 
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Identifying and appropriately training staff are important for fideli- 
ty of program implementation and sustainability of intervention 
effects over time. The guidelines offered in these arenas are practical 
and realistic. 

While each section also mentions the need to integrate the strat- 
egy with other systems and to evaluate the intervention, these topics 
are covered only briefly. The continued emphasis on the need to 
consider maintenance of results after implementation is a plus, as 
more and more funders and Federal agencies are looking for long- 
term sustainability of programs after their initial support runs out. 

To its credit, the Sourcebook also mentions the importance of 
considering cultural and demographic issues pertinent to each inter- 
vention, as well as environmental/contextual concerns when imple- 
menting a particular strategy in a community or school. These are not 
discussed in detail, however. This is unfortunate because these can be 
two significant barriers to successful implementation and sustainabil- 
ity over time. The Sourcebook also could have discussed more fully the 
issue of recruiting high-risk, resistant, mobile famihes into some of 
these types of initiatives and the potential impact on services, infor- 
mation gathering (evaluation), and policy. These criticisms are rela- 
tively minor, however, given the strong and. thorough discussion (rel- 
ative to other guides) of the other content areas discussed. 

Two unique components of the Sourcebook that would be very 
helpful to school or community practitioners are the table on age- 
appropriate violence prevention curricula and a brief fact sheet on 
youth violence. While the statistics on youth violence are constantly 
changing, the fact sheet offers a fi-amework for providing information 
to community leaders, funders, and policy makers to garner their sup- 
port in implementing successful youth violence prevention initiatives. 

(3) Conflict Resolution Education — A Guide to Implementing 
Programs in Schools, Youth Serving Organizations, and 
Community and Juvenile Justice Settings, Office of Juvenile 
Justice and Delinquency Prevention and the U.S. Department 
of Educations Safe and Drug Free Schools, 1996 (#7 in 
Tables) 

This Guide to implementing conflict resolution education (CRE) 
is clear about its focus and target audience. The entire Guide is dedi- 59 
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cated to the description and discussion of implementation of various 
forms of conflict resolution education in schools, communities, and 
juvenile justice settings. So, on the one hand, the Guide offers in-depth 
coverage of a very specific topic and type of intervention strategy, 

. which is consistent with its intended purpose. On the other hand, it 
would not be very helpful to the practitioner interested in imple- 
menting a broad-based, multi-component intervention, unless one 
component of that intervention was conflict resolution education. 

The Guide does a nice job of covering a wide range of topics relat- 
ed specifically to CRE, including the rationale for implementing CRE, 
and the theoretical underpinnings of the different models of CRE: 
process or mediation-based CRE programs and whole-classroom and 
schoolwide implementation models. It was initially exciting to find an 
entire chapter on evaluation listed in the table of contents, only to find 
that the chapter was on the evaluation findings relative to the effec- 
tiveness of CRE rather than information on how to evaluate CRE pro- 
grams. In this regard, the conflict resolution Guide falls into the cate- 
gory of calling for the need to include evaluation^ but providing little 
guidance regarding how to go about it. Another limitation is the lack 
of any discussion about the need to include CRE as a component of a 
more comprehensive violence prevention initiative. Few would argue 
that doing CM alone would be an aidequate long-term preventive 
intervention strategy. 

The same criticism caimot be levied regarding developmental 
appropriateness of the intervention strategy. An entire chapter is 
devoted, albeit mostly in table format, to different abilities and 
processes involved in CRE, broken down into categories including 
early childhood to grade 2, grades 3 to 5, grades 6 to 8, and grades 9 
to 12. A brief chapter is also included on how to establish a conflict 
resolution education program, including discussion of needs assess- 
ment, selection of an appropriate program or curriculum, and selec- 
tion of staff and trainers. A large compendium of contact informa- 
tion, resources, a reading list, and a glossary would be very helpful to 
the practitioner committed to learning about CRE and implementing 
it in a community or school. The CRE Guide was one of the few 
resources to offer specific examples of forms for the various types of 
assessments discussed. Finally, testimonials from school and commu- 
nity providers on the value of CRE, examples of success stories of 
50 implementation and positive outcomes, and summaries of specific 
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projects and sites are scattered throughout the Guide. This informa- 
tion may prove both interesting and helpful to the practitioner look- 
ing beyond the quantitative data and research results to garner sup- 
port for moving forward. 

(4) Early Warning Timely Response: A Guide to Safe Schools, U.S. 

Department of Education, 1998 (#11 in Tables) 

and 

(5) Safeguarding Our Children: An Action Guide, U.S. Departments 

of Education and Justice and American Institutes of Research, 

2000 (#12 in Tables) 

These two resource guides are combined in this review because 
Safeguarding Our Children was specifically developed as the imple- 
mentation guide to Early Warning, Timely Response. Early Warning 
was developed in response to a presidential initiative following the 
school shootings in Eugene, Oregon. As such, its focus was on helping 
educators and others with “what to look for,” the early warning signs 
that relate to violence and other troubling behaviors and “what to do,” 
the “action steps that school communities can take to prevent violence 
and other troubling behaviors,, to intervene and get help for troubled 
children, and to respond to school violence when it occurs.” The 
majority of Early Warning is focused on the “what to look for” issue, 
with a couple of sections at the end initially addressing “what to do.” 
Safeguarding Our Children is fiiUy focused on answering the “what to 
do” question with respect to targeted, at-risk youth who have the 
potential for engaging in violent behavior. 

Both of these guides, while targeting school staff, would be useful 
for anyone working in the area of violence prevention. They are based 
on the premise that school safety plans need to be grounded in 
research-based best practices. A strong underlying theme is that effec- 
tive action plans are strategic, coordinated, and comprehensive. 
Clearly schools do not operate in a vacuum outside of their commu- 
nity and environment. Similarly, support for students needs to occur 
both within the school and outside the school doors in families and 
communities. 

While Early Warninghys out a myriad of risk factors and red flags 
to look out for, there is an appropriate and consistent plea not to use 




the list as a means to label youngsters or to use the warning signs as a 
way to fit children into a specific profile. While the focus for both 
resource guides is on how to identify and intervene with potentially 
violent youth, they place targeted interventions in the context of a 
developmentally appropriate long-term intervention plan that 
includes universal prevention for aU students, not just those at risk for 
violence. Throughout both guides there is also a consistent theme of 
early intervention in a proactive way, rather than waiting to react to 
crises. 

Safeguarding Our Children focuses on creating links between 
teams that address individual student problem behavior (a teacher, 
principal, and mental health professional), a schoolwide team, and a 
larger student support team. While this specific structure may not 
work in aU communities, a strength of the guide is its attention to stu- 
dent mental health needs and the role of such attention in violence 
prevention, as well as the recognition of the need for multiple support 
systems both within and outside tlie school. A brief sidebar outlines 
evaluation criteria to help schools identify a program appropriate to 
their setting. This includes a consideration of outcome evidence, cost, 
implications for personnel and staffing, diversity, and flexibility. Other 
than this brief mention, however, evaluation of program effectiveness 
and staff training issues are not foci of these two resource guides. 

The wealth of practical information contained in these guides is 
usable largely because they are well organized and clearly written. The 
resource and action guides offer a nice example of brevity without 
sacrificing attention to the important details or the need to offer com- 
prehensive coverage of a topic. A particularly nice feature of Early 
Warning is the inclusion of “Action Steps” for students, an “Action 
Planning Checklist,” and a “Crisis Procedure” checklist. Safeguarding 
Our Children also contains detailed descriptions of different types of 
programs and specific guides around developing referral procedures 
and intervening with high-risk youth. 

The last chapter of Safeguarding Our Children discusses many 
pragmatic issues involved in creating and implementing a compre- 
hensive plan, including the need to include community members on 
the schoolwide team and to gather appropriate resources to help 
ensure long-term success. Both guides contain sections of online and 
other Federal resources for more information, as well as specific con- 
tact information on the programs described. 







Of all the resource guides we reviewed, the combination of Early 
Warning and Safeguarding Our Children provided the most compre- 
hensive, easy to read, practicable guide to dealing with potentially vio- 
lent youth in school and community settings. 

( 6 ) Violence Prevention and Safe Schools, UCLA Center for Mental 
Health in Schools, 2000 (#30 in Tables) 

The strength of this guide from the Center’s Clearinghouse on 
Violence Prevention and Safe Schools is the breadth and depth of its 
information on violence prevention, particularly with respect to men- 
tal health issues for young people. Unfortunately this is also its major 
weakness. SpecihcaUy, the packet contains a plethora of information, 
most of it gleaned from other sources and documents. Several pieces 
were reprinted from other resource guides (many contained in this 
review) or downloaded from the Internet. Much of the information is 
independently valuable, but only the Table of Contents connects the 
information in any coherent way. ^ 

The value of the guide in terms of breadth of information and 
variety of resources is diminished by its lack of a coherent strategy for 
its presentation or integration of the material. Many of the resources 
and much of the information are not placed in any context. If this 
were the first guide that a reader came across, and if the reader 
depended on it for help in implementing a violence prevention strat- 
egy, the reader could become easily overwhelmed by the volume of 
different pieces of information. However, if a practitioner had only 
one or two resources available and were looking for lots of other 
resources, the guide might prove helpful. The challenge would be in 
knowing specifically what information was needed (e.g., strategies to 
deal with angry or aggressive youth). The patient reader would also 
need to have the perseverance to look elsewhere for additional infor- 
mation on implementation strategies. 

The guide does provide much useful information. It summarizes 
the American Psychological Association’s recommendations for vio- 
lence prevention, including its recommendations on public policy 
and funding. There is an extensive table on exemplary and promising 
programs reprinted from the Center for Mental Health Services 
(CMHS) and the Center for Substance Abuse Prevention (CSAP). 

The articles on anger control, classroom management with disruptive 53 
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and violent students, bullying, hate crimes, and sexual harassment 
(the latter two topics rarely addressed in any guide) could prove 
extremely helpful to the classroom teacher. There is little guidance, 
however, on what to do with all this information, how to compare or 
prioritize all the different sets of recommendations, or how to medi- 
ate potentially conflicting information on specific issues. 

In sum, Violence Prevention and Safe Schools contains a great deal 
of valuable information, much of it culled from other existing 
resource guides and Federal reports. It is a difficult read, however, 
because there is little rationale for the selection of materials or any 
attempt to organize or integrate overlapping information. For the 
new reader, this guide could be overwhelming. For the experienced 
reader who knows what to look for, it can lead the way to a wealth of 
helpful information in the quest to develop an effective violence pre- 
vention plan. 

(7) The Role of Education in a System of Care: Effectively Serving 

Children with Emotional or Behavioral Disorders (Volume III), 

American Institutes for Research, 1999 (#31 in Tables) 

Volume III in the Center for Effective Collaboration and 
Practice’s Systems of Care series on promising practices in children’s 
mental health focuses on the efforts, experiences, and outcomes of 
three urban sites funded by the Community Mental Health Service 
for Children and Their Families Program (CMHS): South 
Philadelphia, Pennsylvania; East Baltimore, Maryland; and a 
statewide program in Rhode Island. Each of the sites was expected to 
implement a system of care with an emphasis on school-based men- 
tal health services that coordinates with other community partners. 
The volume describes the projects and experiences of each of these 
sites, and also provides some essential cross-site findings and lemons 
from these and other CMHS project sites. One section covers strate- 
gies to overcome potential barriers to implementation. These are 
major strengths of this volume. In particular, the chapter on over- 
coming barriers is filled with practical information on topics not nor- 
mally covered in these resource guides, such as handling teacher con- 
tract issues when teachers are involved in the implementation of an 
intervention or have to participate in significant training. Another 
plus is acknowledgement that building principals largely control what 
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goes on in their school, so upper level administrative support for an 
intervention program, while helpful, is not enough for effective 
implementation in every building. 

The guide focuses on dealing effectively with children who have 
serious emotional or behavioral disorders, the 10 to 15 percent of the 
youth who may account for as much as 80 percent of problems in a 
school. It begins with a thorough review of the literature, both 
research cuid practice based, and a clear rationale for targeting SED 
(Seriously Emotionally Disturbed) youth through a system of care 
that has schools at its core. The approach advocated finds its frame- 
work in the Individuals with Disabilities Education Act (IDEA) and in 
the notion that mental health services should be school based and 
involve collaborative, community-linked services for SED youth. The 
two main principles underlying school-based systems of care are that 
(1) services are child- and family-driven, and (2) cultural competen- 
cy means meeting the needs of diverse clients. 

The guide offers both breadth and depth of coverage of the issues 
regarding intervention with students who have severe emotional or 
behavioral problems in school. It provides very detailed descriptions 
of specific programs, as well as the rationale for advocating for men- 
tal health clinicians in schools. A strength of the effort is its consistent 
support for mental health services as a complement to schoolwide 
preventive interventions as well as the need for multi-system collabo- 
ration and ongoing support for children, families, and teachers. While 
the volume is lengthy, it is generally well organized and easy to read. 
The program descriptions are interspersed with staff observations 
and testimonials about their experiences and their particular role in 
the system of care. Both types of information may be helpful to 
implementing practitioners at other sites. 

While each of these selected sites was in an urban setting, and 
some issues are unique to those settings, many of the lessons learned 
and barriers to overcome would apply to suburban and smaller rural 
communities, because the children face many of the same issues, and 
intervention with SED youth would still require some of the same ele- 
ments of a collaborative system of care between the school system and 
the community. 

If this volume has a weakness, it is in its cursory discussion of staff 
training issues and the lack of much discussion about evaluation of 
intervention outcomes. It should be noted, however, that evaluation 
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of program interventions as a task is not one of the intended goals of 
the volume; rather, the presence of positive program outcome data is 
contained in descriptions of activities and components throughout 
the volume. The volume would be very helpful to the practitioner 
interested in implementing these types of services for SED children, 
but not very helpful to the evaluator charged with setting up a system 
to assess program outcomes. 

The volume includes appendices with program contact informa- 
tion and materials as well as some helpful guidelines for provision of 
in-school wraparound services and some lessons learned (East 
Baltimore site). 

Summary 

There has been a significant increase in recent years in the num- 
ber of publications focusing on school- and community-based youth 
violence prevention and intervention services. Most of the resource 
guides developed focus on a very specific population (e.g.. Seriously 
Emotionally Disturbed or targeted, at-risk youth) or on a specific 
approach (e.g., conflict resolution education). A few resource guides 
attempt to provide a more comprehensive framework for preventive 
interventions while focusing on specific types of best practices (e.g., 
CDCs Sourcebook) or on a specific population of youth (e.g.. Early 
Warning, Timely Response and Safeguarding Our Children). There are 
a few guides that have attempted to review many programs and rate 
their content and applicability to violence prevention (e.g.. Safe 
Schools/Safe Students Guide). There are many new guides available to 
the practitioner charged with developing a violence prevention initia- 
tive. The challenge is to start with the guide, or, in some cases, the 
chapter within a resource guide, that will provide an overview of a 
comprehensive, multi-system, flexible approach to violence preven- 
tion and the rationale and research base for taking such an approach. 
The CDC Sourcebook and Early Warning Action Guide would be par- 
ticularly useful in this regard. The next step would be to examine 
guides that review the types of programs already in place (to deter- 
mine how they might fit into a new model) or guides that examine 
programs that might be productively added to die existing system of 
care (e.g., adding a middle school-focused intervention to strong ele- 
mentary school programs). Another strategy is to seek out a resource 







guide that focuses specifically on a target population or setting or on 
the type of intervention strategy that may be implemented in a school 
or community. 

One weakness of existing guides is their limited discussion of 
evaluation and its role in determining program effectiveness. This 
limitation occurs on two fi^onts. First, there exists a lack of discussion 
of evaluation outcome or process data that can help the reader make 
relative comparisons of program effectiveness or potential when 
choosing from a myriad of possibihties. Of course there are some 
exceptions to this limitation, but even the guides that include evalua- 
tion data as a criterion of assessment are either limited by a general 
lack of good evaluative data on any program (e.g., Safe Schools/Safe 
Students) or have adopted an extremely stringent set of criteria for best 
practices which limits the program options for a community looking 
to implement a new strategy (e.g., Blueprints for Violence Control). 

The second major limitation on evaluation is the lack of attention 
to ways to evaluate a program or set of programs as part of a violence 
prevention initiative. Many resource guides state that evaluation is 
important. Consistently, guides state that evaluative data should be 
used to modify program implementation and can be very valuable to 
potential funders and to the sustainabUity of programs over time, but 
few guides provide any pragmatic guidance for the non-researcher 
about how they might even conduct basic evaluation of their inter- 
vention. A notable exception to this is the Evaluation Handfooofc devel- 
oped by the W.K. Kellogg Foundation ( 1998 ). The Handbook is gen- 
erally well organized and easy to read, and information is presented so 
that someone with only basic knowledge of research or research 
methods could benefit and use the information to address some basic 
evaluative questions or strategies. 
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Table 2: School Violence Resource Guide 
Content Checklist 
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Guide number: Reference to number assigned to 
school violence resource guides on Table 1. 

School Violence Statistics: Inclusion of school 
violence data-to-date. 

Developmental Focus: Attention to age appropriate- 
ness of the information. 

Program Review: Mention of specific school- or 
community- based programs for specific interver^ns 



School-Based Curriculum: Inclusion of curriculum. 
School-Community Partnerships: Steps to take in 
building partnerships. 

Safe School Planning: Steps in planning. 
Evaluation: Inforrnation on performing an evaluation 
of school district projects. (Double check indicates 
in-depth coverage.) 

Agencies/Organizations: References to organizations 
outside of schools that assist in violence prevention. 
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Risk Factors: Specification of factors. 

Protective Factors: Specification of factors. 
Targeted Youth Services: Information on specific 
v^/o/ence prevention strategies for targeted youth. 
(Challenges to Implementation: Discussion of 
barriers to intervention programs. 

Intervention Strategies: Descriptions of strategies. 



Policy implications: Impact of violence prevention 
interventions on policy. 

Staff Training: Information on its Importance. 

Other Resources/References; Inclusion of information 
on additional resources. 

Shaded areas indicate in-depth review of the 
ide in the text Double checks indicate separate 
verage of topic. 
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